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CHAPTER  I 
INTRODUCTION 

Statement  of  Purpose 

It  is  the  purpose  of  this  paper  to  define  the  competitive  computer  selection 
system  used  by  the  United  States  Navy.    Other  selection  systems  are  described,  and 
the  Navy's  system  is  evaluated  in  relation  to  these  other  systems. 

Organization  of  Study 

In  keeping  with  its  expressed  purpose,  this  thesis  is  divided  into  several 
chapters,  each  chapter  provides  the  reader  with  a  logical  element  from  the  total 
package. 

The  competitive  computer  selection  process  used  by  the  Department  of  the 
Navy  is  highly  regulated  and  owes  its  origin  to  the  fact  that  the  Navy  is  a  govern- 
mental agency  with  a  central  office  responsible  for  the  selection  process.     In 
Chapter  II,  the  various  controls  exerted  by  Congress,  the  President,  the  Department 
of  Defense,  and  the  Secretary  of  the  Navy  are  examined.    The  responsible  central 
office  is  a  very  significant  factor  shaping  the  selection  system.    Accordingly,  the 


mission  and  organizational  relationships  of  this  office  are  defined. 

Chapter  III  examines  the  details  of  the  selection  procedures.    The  total   cycle 
of  acquiring  a  computer  system  involves  two  distinct  and  separate  phases.     First, 
the  user  conducts  a  system  study  to  define  the  management  data  system  and  prove 
its  soundness.    The  second  part  concerns  the  procurement  actions  necessary  to 
bring  a  computer  system  on  board.    This  paper  will  address  only  the  latter  part  of 
the  cycle  and  thus  assumes  that  a  good  system  study  has  been  conducted  and  the 
prospective  user  therefore  knows  his  true  systems  requirements.     Chapter  III  is 
therefore  directed  toward  the  five  major  activities  that  involve  the  central  selection 
office.    The  five  activities  are:    Specification  Review,  Preparation  of  the  Request  for 
Proposal  and  Selection  Plan,  Vendor  Liaison  and  Validation  of  Proposals,  Evaluation 
of  Proposal,  and  Contract  Negotiation  and  Award. 

Chapter  IV  discusses  the  results  of  a  survey  conducted  by  Dr.  Norman 
Schneidewind.    This  survey  purports  to  show  four  different  catergories  of  users,  and 
what  practices  are  used,  to  select  computer  systems,     hi  addition,  the  validation 
techniques  of  computer  simulation  and  hand  timing  arc  discussed.    The  latter  part 
of  this  chapter  is  directed  toward  two  alternative  evaluation  systems- -the  Cost- 
Effectiveness  and  the  Weighted  Scoring  methods. 

Chapter  V  evaluates  each  of  the  five  major  activities  in  the  Navy  selection 
process  in  relation  to  the  methods  described  in  Chapter  IV.    Chapter  VI  summarizes 
the  findings  of  the  study. 


Statement  of  Research  Question 

This  thesis  provides  answers  to  the  following  questions: 

o  What  is  the  relationship  of  the  Navy's  system  to  alternative  systems  that 

might  be  used? 
o  What  are  the  significant  features  of  the  competitive  computer  selection 

used  by  the  United  States  Navy? 
©  What  alternative  methods  could  be  used? 

©  What  are  the  strong  points  in  the  Navy's  system? 

o  What  are  the  controls  imposed  by  higher  authority? 

o  Who  in  the  Navy  is  responsible  for  the  selection  process? 


CHAPTER  II 
THE  GOVERNMENT  ENVIRONMENT 

The  system  the  United  States  Navy  uses  to  competitively  select  computers  is 
a  result  of  the  interaction  of  many  and  diverse  influences.    There  are  strong- 
pressures  exerted  on  the  Navy  by  such  external  forces  as  the  United  States  Congress, 
the  President  of  the  United  States,  the  Bureau  of  the  Budget,  the  Department  of 
Defense  and  many  others.     In  addition,  there  are  active  forces  within  the  Navy 
itself.    The  most  significant  action  element  in  the  Navy  is  the  Automatic  Data 
Processing  Equipment  Selection  Office.    The  purpose  of  this  chapter  will  be  to 
define  these  influences.    The  effect  of  these  forces  will  be  examined  in  Chapter  III. 

Congressional  Influence 

The  most  significant  influence  exerted  by  the  Congress  is  Public  Law  89-306 
enacted  on  October  30,  1965.    This  law  is  known  as  the  Brooks  Bill  because  the 
legislation  is  a  product  of  the  Government  Activities  Subcommittee  of  the  House 
Committee  on  Government  Operations,  Congressman  Jack  Brooks  is  chairman 


of  this  subcommittee.      The  need  for  this  legislation  was  defined  by  the  subcommittee 

in  its  report  on  H.  R.  4845: 

Since  1958,  up  to  the  time  of  the  hearings  on  H.  R.  4845,  the  GAO  has  issued 
approximately  100  audit  reports  to  agencies,   congressional  committees,  and  to 
Congress  revealing  serious  shortcomings  in  the  manner  in  which  specific  agencies 
acquired  and/or  utilized  ADP  equipment.    The  major  deficiencies  cited  in  these 
reports  have  been: 

(a)  Inadequate  feasibility  studies; 

(b)  Uneconomical  and  ineffective  equipment  utilization; 

(c)  Overpayments  resulting  from  inadequate  management  practices;  and 

(d)  Uneconomical  procurement  of  equipment. 

Based  on  this  need, -the  Brooks  Bill  purports  to  satisfy  three  vital  management 
needs:    (1)  more  adequate  management  information,    (2)  optimum  utilization  of  ADP 
(Automatic  Data  Processing)  equipment  through  sharing,  and    (3)  multiple  use  and 
economic  acquisition.    The  subcommittee  indicated  that  an  economical  acquisition 
program  involved  three  principal  factors: 

(a)  Improving  the  Government's  bargaining  position  through  volume  acquisition; 

(b)  Basing  lease  versus  purchase  evaluations,  whenever  possible,  on  the  long- 
term  value  of  the  equipment  to  Government  as  a  whole;  and 

(c)  Selecting  that  equipment  for  purchase  which,  on  a  Government- wide  basis 
Offers  the  largest  purchase  advantage.  3 


U.S.,  Congress,  House,  Committee  on  Government  Operations,  Automatic 
Data  Processing  Equipment,  Hearings,  before  a  subcommittee  of  the  Committee  on 
Government  Operations,  House  of  Representatives,  onH.R.  4845,  89th  Cong.  ,   1st 
sess.  ,   1965,  p.  II. 

2H.R.   Rep.  No.  802,  89th  Cong.  ,   1st  sess.  ,   1965,  p.   17. 

3lbid.  ,  p.  23. 


To  satisfy  the  needs  of  management  as  cited  above,  Public  Law  89-306 
increased  the  authority  and  responsibility  of  three  federal  agencies:    the  General 
Services  Administration,  the  Bureau  of  the  Budget,  and  the  National  Bureau  of 
Standards  in  the  Department  of  Commerce. 

The  Bureau  of  the  Budget  is  charged  with  the  fiscal  and  policy  control  of  ADP 
management  of  the  government.    The  bill  provides  that  the  actions  of  any  agency 
taken  under  the  authority  of  the  law  are  subject  to  the  review  and  approval  of  the 
Bureau  of  the  Budget. 

The  Administrator  of  the  General  Services  Administration  (GSA)  is  charged 
with  the  primary  responsibility  for  coordinating  die  government's  ADP  management, 
subject  to  the  policy  and  fiscal  control  of  the  Bureau  of  the  Budget.    The  law 
specifically  directs  the  administrator  to  "coordinate  and  provide  for  the  economic 
and  efficient  purchase,  lease,  and  maintenance  of  automatic  data  processing 
equipment  by  Federal  agencies. "4    The  bill  however  does  limit  this  authority.     Each 
agency  was  given  the  latitude  to  continue  determining  what  their  ADP  requirements 
were. 

The  National  Bureau  of  Standards  is  responsible  for  the  technical  aspects  of 
this  coordinated  management  program.  The  Bureau  is  responsible  for  conducting 
research  and  development  programs  based  on  the  needs  of  the  government.    The 


Federal  Property  and  Administrative  Services  Act  of  1949  (63  Stat.   377)  as 
amended,  Sec.    111. (a). 


drafters  of  the  bill  intended  for  the  Bureau  of  Standards  to  direct  its  attention  toward 
the  problem  of  lack  of  equipment  compatability.    The  intent  of  the  law  however  was 
not  to  authorize  the  Bureau  to  launch  a  broad  research  and  development  program. 
Rather,  the  intent  was  aimed  at  supplementing  the  government  research  effort  in 
coordination  with  other  federal  agencies  and  monitoring  developments  in  the  industry 
affecting  areas  covered  by  the  law.    A  copy  of  Public  Law  89-306  is  shown  in 
Appendix  A. 

Presidential  Influence 


President  Lyndon  B.  Johnson  on  June  28,   1966,   sent  a  memo  to  the  heads  of 
all  federal  agencies.    The  purpose  of  the  memo  was  to  stimulate  action  to  improve 
the  management  of  the  ADP  resources  in  the  government.    The  President  referred 
to  the  recognition  that  the  Congress  had  given  to  this  problem  by  enacting  the  Brooks 
Bill.    President  Johnson  pointed  to  the  fact  that  the  federal  government  was  using 
2,600  computers,  employing  71,000  people  in  the  field,  and  spending  over  $2  billion 
annually  to  acquire  and  operate  ADP  equipment.    The  President  said:    "I  want  the 
head  of  every  Federal  agency  to  explore  and  apply  all  possible  means  to  use  the 
electronic  computer  to  do  a  better  job  .   .    .  manage  computer  activity  at  die  lowest 
possible  cost.  "5    The  President  pointed  to  four  specific  areas  that  were  to  be 


^Lyndon  B.  Johnson,   Memorandum  to  Heads  of  All  Federal  Agencies,   28  June 
1966. 


given  priority  attention:   better  methods  of  procuring  >ADP  systems;  achieving 
better  utilization  of  existing  capabilities  through  sharing  and  joint  use  arrangements; 
re-utilizing  excess  equipment  when  possible;  and  achieving  greater  compatability. 

The  President  indicated  to  the  Heads  of  the  Agencies  that  he  expected  them  to 
cooperate  fully  with  the  Bureau  of  the  Budget,  the  General  Services  Administration, 
and  the  Department  of  Commerce  in  accomplishing  these  actions.  (See  Appendix  B 
for  a  copy  of  the  President's  memorandum.) 

^Secretary  of  Defense  Influence 

Secretary  McNamara  acted  upon  the  President's  memorandum  with  a  statement 
of  policy  to  the  top  management  of  the  Department  of  Defense  (DOD).  °   The  Secretary 
indicated  his  desire  to  attain  the  objectives  set  down  by  the  President  but  further  a 
desire  to  set  an  example  to  the  rest  of  the  government  agencies.    The  memorandum 
pointed  to  the  fact  that  DOD  operated  2,  000  of  the  2,600  computers  cited  in  the 
President's  memo  and  employed  51,000  of  the  71,000  federal  employees  engaged  in 
the  ADP  effort.    It  was  therefore  the  view  of  the  Secretary  of  Defense  that  DOD  had 


Robert  Sc  McNamara,  Memorandum  to  The  Secretaries  of  the  Military 
Departments,  The  Chairman  of  the  Joint  Chiefs  of  Staff,  The  Director  of  Defense 
Research  and  Engineering,  The  Assistant  Secretaries  of  Defense,  The  Director  of 
Defense  Agencies,  29  July  1966. 


to  bear  a  large  share  of  responsibility  for  improved  management  of  the  governments 
ADP  resources. 

The  Secretary's  memorandum  defined  the  areas  that  required  management 
attention,,    There  are  several  items  of  specific  interests  in  this  paper: 

(a)  We  must,  prior  to  computer  selection,  develop  and  issue  systems 
specifications  which  adequately  describe  the  systems  to  be  performed  and  which 
will  result  in  selection  of  computers  which  can  satisfy  the  requirements  of  that 
specification.    We  should  not  be  forced  to  acquire  additional  units  at  later  dates 
or  replace  the  computers  prematurely  because  of  selections  based  on  inadequate 
system  specifications  or  on  less  than  responsive  vendor  proposals.    Military 
Departments  and  DOD  agencies  should  centrally  prescribe  and  control  the 
development  of  systems  specifications  for  computer  acquisition  purposes. 

(b)  We  must  make  competitive  computer  evaluation  and  selection  a 
professional  endeavor  and  organize  and  staff  for  it  accordingly.     Staffs  performing 
this  function  for  the  senior  automatic  data  processing  policy  officials  should  be 
divorced  from  computer- using  organizations.     Steps  should  be  taken  to  assure 
that  these  staffs  are  the  sole  point  of  contact  with  the  computer  industry  on  all 
matters  pertaining  to  computer  selection  for  their  respective  agencies. 

(c)  ...  Further,  computers  will  not  be  selected  until  the  performance  of  the 
complete  hardware/ software  package  required  in  the  systems  specification  and 
request  for  proposal  has  been  clearly  demonstrated  by  either  a  full-scale  or 
bench  mark  test.    (Emphasis  supplied.)  ' 


Secretary  of  the  Navy  Action 

The  Secretary  of  the  Navy  initiated  a  study  effort  within  the  Department  of  the 
Navy  to  define  the  actions  that  should  be  taken  to  satisfy  the  policy  statements  of 
President  Johnson  and  Secretary  McNamara. 


7Ibid. 
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Aii  instruction  signed  by  Secretary  of  the  Navy  Paul  H„  Nitze  on  June  30,  1967 

established  the  Automatic  Data  Processing  Equipment  Selection  Office  (ADPE  SO). 

The  mission  assigned  to  this  office  is: 

a.    To  evaluate  and  select  for  approval  by  the  Special  Assistant  to  the 
Secretary  of  the  Navy  (SASN),  automatic  data  processing  equipment  to  be 
acquired  by  the  Department  of  the  Navy,  except  as  defined  in  paragraph  4 
below  o 

b0    To  act  as  the  Department  of  the  Navy  contracting  office  for  the 
procurement  of  ADPE„ 

The  exception  mentioned  above  relates  to  equipments  integrated  with  weapons 

systems,  punch  card  equipment,  and  other  special  situations  exempted  by  SASN. 

The  functions  assigned  ADPESO  are:     • 

(a)  Serve  as  the  principal  point  of  contact  with  the  industry  on  matters 
concerning  validation,  evaluation,   selection,  and  procurement  of  ADPE  for  the 
Department  of  the  Navy. 

(b)  Develop  and  administer  for  the  Department  of  the  Navy  plans,  policies, 
procedures  and  methods  governing  ADPE  specification  preparation,  proposal, 
solicitation,  proposal  evaluation  and  selection. 

(c)  To  solicit  sources,  negotiate  and  award  contracts,  place  delivery  orders 
against  Federal  Supply  Schedules  and  administer  contracts  and  orders  for  such 
ADPE  acquisitions. 8 

The  Director  of  the  Automatic  Data  Processing  Equipment  Selection  Office, 

reports  to  the  Special  Assistant  to  the  Secretary  of  the  Navy.    Thus  he  is  very  high 


^U.S.  ,  Department  of  the  Navy,  "Automatic  Data  Processing  Equipment 
Selection  Office;  establishment  of,"    Secretary  of  the  Navy  Instruction  5430.81, 
Department  of  the  Navy,  Washington,  D.C.  ,  20390,  June  30,   1967. 
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iii  the  Navy  department  hierarchy  and  is  separated  from  computer- using  organizations. 
(See  Appendix  D  for  a  diagram  of  this  organizational  alignment.) 

ADPESO  Action 


Upon  completion  of  the  first  milestone  of  organizing  and  staffing,  management 
turned  and  directed  it  attention  toward  the  mission  assigned  to  ADPESO.     In  order  for 
this  to  be  done,  the  methods  of  operation  had  to  be  defined  and  approved.    This  effort 
required  ADPESO  personnel  to  gather  the  various  directives  and  regulations  that 
apply  to  ADP  management  in  the  government.    There  is  a  large  number  of  directives 
toward  this  end.    (See  Appendix  E.)    For  the  purposes  of  this  paper  however,   only 
the  most  significant  ones  will  be  discussed. 


Bureau  of  the  Budget  Circular  No.  A- 54: 

Policies  on  Selection  and  Acquisition  of  Automatic  Data  Processing  Equipment 


The  purpose  of  this  directive  is  to  prescribe  the  government's  policy 
concerning  ADP  equipment  selection  and  provide  guidance  relative  to  the  method  of 
acquisition- lease,  purchase  or  a  lease  with  an  option  to  purchase  method. 

The  government's  policy  concerning  selection  specifies:    that  all  selections 
will  be  made  based  on  a  set  of  systems  specifications;  that  selection  procedures  will 
assure  equal  opportunity  for  all  manufacturers  to  participate;  that  the  two  prime 
factors  to  be  considered  in  the  selection  process  are  (1)  the  equipment's  ability  to 


12 


fulfill  the  systems  specification,   and  (2)  its  overall  costs  including  those  applicable 
to  acquisition,  preparation  for  use  and  operation. 

The  agency  in  question  is  obliged  to  select  that  method  of  ADPE  acquisition 
which  offers  the  greatest  advantage  to  the  government.    The  purchase  method  is  to 
be  used  when  all  the  following  conditions  exist: 

o  There  is  a  reasonable  expectancy  that  the  ADPE  in  question  can  be 

successfully  and  advantageously  used; 
©  That  after  a  comparative  cost  analysis  has  been  conducted,   it  is 

established  that  a  cost  advantage  can  be  obtained  in  six  years  or  less 
and  the  cost  of  moncy9  has  to  be  used  in  conducting  this  cost  analysis; 
©  The  ADPE  will  be  able  to  satisfy  the  requirements  of  the  system  beyond 

the  point  where  the  purchase  method  provides  a  cost  advantage;  and 
o  The  feasibility  and  economics  of  performing  equipment  maintenance  with 

in-house  resources  has  been  considered. 
The  lease-with-  option-to-purchase  method  is  to  be  used  when  it  is  necessary 
or  advantageous  to  acquire  the  equipment,  but  it  is  preferable  to  defer  temporarily 
the  decision  to  purchase  when  circumstances  do  not  fully  satisfy  the  conditions 
which  would  indicate  purchase. 


9 

The  procedure  used  is  defined  by  Bureau  of  the  Budget  Circular  A- 94.    A 

sample  problem  from  this  circular  is  shown  as  Appendix  J. 
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The  lease  method,  without  option  to  purchase,   should  be  used  when  it  is 
advantageous  or  necessary  to  acquire  the  equipment  and  when  it  has  been  determined 
that  anyone  of  the  prerequisite  conditions  of  purchase  cannot  be  attained. 


Department  of  Defense  Directive  4105.  62: 
P r oposal.  Evaluation  and  Source  Selection 


The  purpose  of  this  directive  is  to  establish  the  objectives,  principles,  and 
policy  for  the  evaluation  .of  proposals  and  the  selection  of  contractual  sources.    The 
directive  defines  three  very  significant  parties  and  their  roles  in  the  selection 
process. 

First  is  the  Source  Selection  Authority  (SSA).    He  is  an  official  designated  to 
direct  the  source  selection  process  and  to  make  die  source  selection  decision.    The 
principal  duties  of  this  official  are: 

1.  Designate  the  chairmen  and  the  membership  of  the  Source  Selection 
Advisory  Council. 

2.  Establish  guidelines  for  the  actions  to  be  taken  by  the  Source  Selection 
Advisory  Council  and  die  Source  Selection  Evaluation  Board. 

3.  Make  the  decision  as  to  the  source,  after  an  in-depth  review  and 
consideration  of  all  information  and  data  available  from  the  Source  Selection 
Advisory  Council  and  die  Source  Selection  Evaluation  Board. 
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Second  is  the  Source  Selection  Advisory  Council  (SSAC).    It  is  composed  of  a 
group  of  senior  military  and  civilian  personnel  and  is  appointed  by  the  SSA  and  acts 
as  his  staff  and  advisors  in  the  source  selection  process.,    The  principal  duties  of 
this  group  are: 

1.  Establish  the  evaluation  criteria  and  assign  weight  factors  to  each  item 
using  the  recommendations  of  the  Selection  Plan  which  was  prepared  by  the  System/ 
Project  office,  in  this  case  ADPESO. 

2.  Designate  the  chairman  and  members  of  the  Source  Selection  Evaluation 
Boar  du 

3c     Review  the  findings  of  the  Source  Selection  Evaluation  Board. 

Third  is  the  Source  Selection  Evaluation  Board  (SSEB),  This  is  a  group  of 
military  and  civilian  personnel,  from  the  various  functional  and  technical  areas. 
The  duties  of  this  group  are: 

1.  Evaluate  and  score  the  proposals  submitted  by  the  bidders;  and 

2.  Prepare  a  narrative  justification  for  the  evaluation  results. 

This  directive  thus  establishes  a  three-level  review  process  for  the  evaluation 
of  system  proposals. 

Based  on  these  various  directives  and  the  knowledge  and  expertise  of  the  staff 
of  ADPESO,  the  Navy's  computer  selection  process  was  developed.    While  the  many 
directives  provided  broad  policy  guidelines,  detail  methods  or  techniques  were  not 
specified. 
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The  method  of  evaluating  the  proposals  is  based  on  a  system  called  the 
"Cost- Value  Technique"  from  a  book  by  Edward  O.  Joslin,  Computer  Selection.  10 
Mr.  Joslin  wrote  this  book  while  attending  Boston  College  and  working  for  EDP 
Equipment  (Selection)  Office  of  the  U.S.  Air  Force.     Currently  he  is  Head  of  the 
Techniques  Development  and  Analysis  Division  at  ADPESO. 

The  next  chapter  will  treat  the  details  of  the  Navy's  Competitive  Computer 
Selection  System. 


uEdword  O.  Joslin,  Computer  Selection,  (Reading,  Massachusetts:    Addison 
Wesley  Publishing  Company,  Inc.,    1968),  p.  20. 


CHAPTER  III 
U.S.    NAVY  COMPETITIVE  COMPUTER  SELECTION  SYSTEM 

The  Navy's  competitive  computer  selection  system  starts  when  the  system 
specifications  for  a  particular  data  system  are  sent  to  the  ADPESO  for  action.    The 
actions  taken  by  ADPESO  can  be  arranged  into  five  steps: 

Specification  Review, 

Preparation  of  the  Request  for  Proposal  (RFP),and  Selection  Plan, 

Vendor  Liaison  and  Proposal  Validation, 

Proposal  Evaluation,  and 

Negotiation  and  Contract  Award. 
Each  of  these  areas  will  be  discussed  and  explained. 

Specification  Review 

The  ADPESO  actually  performs  three  different  reviews  of  each  set  of 
specifications.    The  purpose  of  the  first  is  to  determine  if  the  user's  requirement 
can  be  satisfactorily  satisfied  by  sharing  a  computer  system  currently  operative  in 
some  other  area,  or  by  reutilization  of  some  system  not  in  use  or  scheduled  to 
become  idle  and  owned  by  the  government    The  second  review  is  conducted  to 
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determine  whether  this  requirement  should  be  satisfied  by  a  competitive  or  a  sole 
source  procurement  action.    While  these  two  reviews  are  important  they  are  not 
germane  to  the  subject  of  this  paper;  accordingly  attention  will  be  devoted  to  the 
third  review  only. 

The  third  review  performed  by  ADPESO  is  concerned  with  the  data  contained  in 
the  system  specifications.  1    Three  parts  of  the  user's  requirements  are  analyzed. 
First,  the  justifications  provided  for  conditions  stated  as  mandatory  are  checked. 
Second,  the  review  examines  the  rationale  provided  for  the  requirements  described 
as  desirable  features.    The  last  part  of  this  review  is  concerned  with  the  workload 
description. 

Justification  of  Mandatory  Requirements 

A  mandatory  requirement  is  defined  as,  "those  requirements  which  the  supplier 
must  satisfy  to  be  considered.  "2    The  Navy's  policy  is  that  the  mandatory  requirements 
shall  not  be  stated  in  a  manner  that  will  restrict  the  free  competition  among  all  of 
the  qualified  bidders.     Further  the  requirements  cannot  be  oriented  toward  any 
specific  vendor's  equipment,   software,  or  services. 


■MLJ.  S.  ,  Department  of  the  Navy,   Specification,  Selection,  and  Acquisition  of 
Automatic  Data  Processing  Equipment,   Secretary  of  the  Navy  Instruction  10462.  13, 
Department  of  the  Navy,  Washington,  D.C.  20390,  p.  B-2.    The  format:  for  the 
systems  specifications  is  shown  in  Appendix  F. 

2 Ibid.  ,  p.  B-ll. 
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The  specifications  must  be  reviewed  to  insure  that  the  user  has  justified 
all  mandatory  requirements,  and  that  the  number  of  them  is  held  to  a  minimum.     In 
addition  the  reviewer  must  insure  that  the  requirement  does  not  restrict:  free 
competition  in  any  way.    The  justification  for  any  mandatory  item  has  to  be  expressed 
in  terms  of  "  .   .   .  why  alternative  approaches  would  not  prove  satisfactory.    1  nis 
justification  should  show  that  the  cost  of  effectively  satisfying  the  requirement  in 
any  alternate  way  would  be  prohibitive,  or  that  the  required  effectiveness  can  only 
be  satisfied  in  the  way  shown.  "3    A  special  check  is  also  made  to  ensure  that 
mandatory  supplier  capabilities,  i.e.,   system  maintenance,  training,  and  conversion, 
etc.  ,  are  not  confused  with  mandatory  system:  requirements.    An  example  of  some 
mandatory  requirements  and  the  types  of  justification  required  is  shown  in  Figure  1. 

Following  are  examples  of  requirements  that  could,  if  properly  justified,  be 
mandatory.  * 

o  Equipment  Availability 

If  the  equipment  must  be  installed  and  operational  by  a  specific  calendar 
date  to  meet  the  requirement  of  the  system(s),  the  user  indicates  the  latest  acceptable 
operational  date.    It  should  be  noted  that  to  specify  a  date  is  an  exception  to  the 
normal  practice. 


3 Ibid.  ,  p.  B-2. 
4Ibid.  ,  p.  B-ll. 
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Mandatory  Requirements 


Justification 


a.    Supplier  Support  -  The  supplier 
must:  agree  to: 

(1)    Train  or  provide  training  for, 
system  analyst,  programmers, 
and  operating  personnel. 


(2)    The  maintenance  personnel 
provided  must  have  a  SECRET 
Security  Clearance. 

b.    Workload  Demonstration  -  The 
benchmarks  must  be  successfully 
run  in  5.00  hours  or  less. 


c. 


COBOL  Compiler  -  The  COBOL 
Compiler  proposed  and  demonstrated 
with  this  system  must  be  a  USASI 
Standard  COBOL  Compiler  (X3.  23- 
1968), 


(1)  The  supplier  must  at  least  be 
responsible  for  the  training  to 
assure  that  the  training  is  of  the 
proper  type.     If  this  training  does 
not  provide  the  proper  information 
the  system,  cannot  be  properly 
utilized. 

(2)  This  is  necessary  to  meet  local 
security  requirements.     See 
paragraph 

Completion  of  the  benchmarks  is 
required  to  demonstrate  the  timing 
capability  of  the  proposed  system. 
Failure  to  complete  the  benchmarks 
in  less  than  5.  00  hours  would  indicate 
that  the  supplier's  equipment  could  not 
complete  the  activity's  projected  work- 
load in  the  600  hours  a  month  available. 
Completion  of  the  benchmarks  will  a 
also  provide  the  validation  of  the 
existence  of  a  suitable  executive  system 
and  compiler. 

This  is  required  for  compliance  with 
Department  of  Defense  acceptance  of 
the  USASI  Standard. 


SOURCE:    U.  S.  ,  Department  of  the  Navy,  Specification,   Selection,  and 
Acquisition  of  Automatic  Data  Processing  Equipment,   Secretary  of  the  Navy 
Instruction  10462. 13,  "bcpalrlmcnt  of  thel^avy",~Wn": iiu lgton,  D.C.   20390,  p.  D-12. 


Figure  1  —  Mandatory  Requirements  and  Their  Justification 
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o  Special  Input/Output  Requirements 

If  special  or  nonstandard  equipment  must  interface  with  the  required 
equipment,  or  if  some  special  characteristics  are  required,   such  as  special 
character  sets  for  the  printer,  minimum  print  characters  per  line,  etc.  ,  the 
requirement  must  be  stated  in  such  detail  that  the  vendors  will  understand  it. 

©  Operational  Requirements 

The  user  would  define  here  any  requirement  that  exists  for  a  reserve 
for  planned  expansion,  mobilization,  and  unavoidable  workload  peaking.    Additionally, 
any  requirement  that  all  or  certain  portions  of  processing  must  be  accomplished 
within  a  certain  time  frame  are  to  be  indicated.     If  the  system  is  a  real-time  system 
the  user  can  indicate  the  response  time  required.     If  the  user  has  a  peak  load  that 
must  be  completed  during  a  specific  time  frame,  the  requirement  must  be  stated 
in  such  sufficient  detail  that  the  suppliers  can  properly  evaluate  the  situation. 

o  Communications 

The  user  may  have  special  communications  requirements,   such  as 
remote  stations,  interface  with  communications  network  terminals  or  tributary 
terminal  devices,  etc.     If  this  is  the  case  then  the  requirements  must  be  defined  so 
the  vendors  can  respond  properly. 

o  Security  Requirement 

This  is  particularly  applicable  to  Department  of  Defense  activities.     If 
the  user's  system  docs  involve  national  security  matters  the  rules  applicable  to  the 
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supplier  and  the  proposed  installations  must  be  stated.     Jn  addition  the  applicable 
regulation  must  be  cited. 

©  Expansion  Capability 

The  user  should  state  the  expansion  capability  required  of  the  system  for 
each  year  of  the  systems  life.    This  is  specified  as  a  percentage  increase  over  the 
first  year,  or  by  increased  yearly  extension  factors. 

There  are  other  mandatory  conditions  which  might  be  specified.    Normally  the 
user  will  state  the  amount  of  time,  per  month,  that  is  acceptable  for  completion  of 
the  workload.    This  is  determined  based  on  the  time  required  to  complete  the 
benchmark.    The  determination  process  will  be  reviewed  later. 

The  experience  to  date  indicates  that  most  conditions  claimed  as  mandatory 
are  not  usually  sufficiently  justified,   so  the  requirement  is  classified  as  a  desirable. 
It  is  also  possible  that  as  the  threshold  or  limit  changes,  the  condition  can  be 
treated  as  a  mandatory  requirement  under  one  set  of  criteria  and  also  as  a 
desirable  feature  using  a  different  criterion;  for  example,  it  might  be  a  mandatory 
requirement  that  the  supplier  install  the  proposed  equipment:  not  later  than  a  given 
date,  and  desirable  for  it  to  be  installed  earlier. 


E.O.  Joslin,  "Competitive  Computer  Selection  Within  the  Department  of  the 
Navy,"    (unpublished  paper  presented  to  the  Diebold  Group,   Lie,  October,   1969. 
Mr.  Joslin  is  Head,  Techniques  Development  and  Analysis  Division  of  ADPESO). 
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Rationale  For  Desirable  Features 


A  desirable  feature  is  defined  as  "those  features,  characteristics,  and 
services  which  the  user  desires,  but  which  arc  not  mandatory  for  a  supplier  to 
provide  to  be  considered  as  meeting  the  overall  requirement. "o 

For  each  desirable  feature  specified  by  the  user,  a  statement  of  value  has  to  be 
given  as  a  justification.    The  value  has  to  be  stated  in  terms  of  dollars  over  the  life 
of  the  system.    However,  the  rationale  required  to  define  a  desirable  condition  is  less 
rigid  than  that  required  of  a  mandatory  item.    A  review  of  statements  of  desirable 
features,  will  attempt  to  find  two  characteristics.     First:    is  the  item  adequately 
described  and  second  does  the  rationale  show  why  the  item  is  desirable.    The  only 
acceptable  explanation  of  desirability  is  one  expressed  in  dollar  terms.     In  fact  when 
possible  the  user  is  suppose  to  provide  two  value  or  worth  figures.    One  figure 
defines  the  savings  or  cost  avoidance  that  will  be  realized  if  the  feature  is  provided. 
The  second  figure  provides  the  cost  of  achieving  this  desirable  in  any  possible 
alternative  way.    The  lower  of  these  two  figures  is  the  value  assigned  to  that  feature. 

Following  is  a  list  of  the  type  items  which  may  be  listed  as  desirable  features.  7 

©  Equipment  Availability 

The  user  can  indicate  the  desirability  of  having  the  equipment  installed 
as  much  as  three  months  in  advance  of  the  required  delivery  date. 


6SccNavInst  10162.  13,  p.  B-ll. 
7Ibid.,  p.  B-ll. 
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o  Software 

If  the  user  desires  any  software  packages  to  be  provided  by  the  vendor, 
each  one  is  described.     Some  examples  are:    report  generators,  sort  routines, 
scientific  subroutines  (such  as  general  factor  analyses,  multi- discriminant  analyses, 
etc.),  and  PERT  cost  and  time. 
o  Supplier  Support 

The  user  can  specify  items  of  support  desired  of  the  supplier  such  as, 
training  to  be  conducted  on-  site,   some  number  of  man- months  of  analyst  support  in 
conversion  of  programs  to  the  new  system,  and  on-site  maintenance. 
o  Mathematical  Computation 

Special  mathematical  capability  can  be  requested  such  as  the  ability  to 
handle  floating  point  computation. 
o  Utilization  Log 

The  user  can  request  that  the  equipment,   in  conjunction  with  software, 
provide  a  daily  utilization  report  by  job  number. 

Figure  2  shows  some  examples  of  desirable  features  and  die  corresponding 
value  statements. 

Statement  of  Workload 

During  the  review  of  the  mandatory  and  desirable  features,    the 
reviewer    must  also  concern  himself  with  the  workload  definition.    This  is  so 
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Desirable  Features 


Value  Statement 


1.    Flow  charting 
program  for 

COUOL 


Expansion 
capability  of  up  to 
fifty  per  cent  over 
the  stated  life 
world  oad 


It  is  estimated  that  a  potential  cost  avoidance  of  $60,  000 
would  accrue  to  the  government  with  the  use  of  a  Flow 
chart  preparation  program.    This  is  based  upon 
programmers  devoting  5  per  cent  of  their  time  to  the 
task  of  flow  charting.    A  staff  of  thirty  earning  an 
average  of  $9,000  annually  for  four  years  is  $1,080,000. 
Five  per  cent  of  this  figure  is  $54,000.    However,   it  is 
estimated  that  this  would  cost  528  hours  of  computer 
time  over  four  years.    At.  $37.  50  per  hour  (non-peak 
time  value),  this  is  $19,800  total  cost.  Therefore,  a  net- 
savings  of  $34,  200  will  result.    An  alternate  source  of 
supply  for  such  a  program  indicates  a  cost  of  $8,000. 
This  cost  was  obtained  from  quotes  received  from 
private  industry  suppliers.    Therefore,  this  alternate 
source  cost  of  $8,  000  is  taken  to  be  the  value  of  this 
program. 

A  system  which  would  require  the  full  five  hours  to  meet 
the  mandatory  requirement  for  completing  the 
benchmarks  would  have  the  required  life  of  five  years 
and  is  expected  to  cost  $1,000,000.    However,  the  stated 
system  life  is  based  on  expansion  of  the  workload  which 
has  already  been  assigned  to  this  organization.     In 
addition  to  the  capability  to  handle  the  known  workload, 
it  would  be  desirable  to  be  able  to  handle  two  additional 
projects  which  have  a  fifty  per  cent  chance  of  being 
assigned  to  this  organization.    The  first  project  (project 
XYZ)  would  require  an  additional  capability  of  approxi- 
mately thirty  per  cent  of  that  required  for  the  stated 
workload.    The  second  project  (project  ABC)  would 
require  an  additional  time  capability  of  twenty  per  cent 
and  would  also  require  the  availability  of  a  remote  tern 
terminal.    Our  ability  to  handle  project  XYZ  could  save 
an  ultimate  expenditure  of  $300,000  over  a  three  year 
period  in  contracted  computer  time.     Likewise,  the 
ability  to  handle  project  ABC  could  result  in  savings  of 


Figure  2  —  Statements  of  Value 
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Desirable  Features 

Value  Statement 

3.  Supplier  provided 
on-site  training  of 
sixty 

programmers, 
thirty  analysts, 
and  ten  operators 

4.  On-site 
maintenance 

$200,000.    However,  because  there  is  only  a  fifty  per 
cent  chance  of  these  projects  being  assigned  to  this 
organization  only  half  of  the  possible  $500,000  saving,  or 
$250,000  is  being  stated  as  the  value  of  this  expansion 
capability. 

If  this  training  is  not  provided  by  the  supplier,  a  cost  of 
approximately  $20,000  would  be  incurred  in  hiring  a 
consultant  to  supply  this  training  in  accordance  with  the 
supplier's  direction.    If  this  training  is  not  accomplished 
on-site,  an  additional  cost  of  approximately  200  man- 
weeks  (assuming  two  week  courses)  of  per  diem  or 
$19,200  plus  travel  cost  could  be  incurred. 

If  on- call  maintenance  is  all  that  is  offered,  this  would 
mean  one  or  two  hours  of  additional  lost  time  per  call, 
and  with  die  expected  five  calls  a  month,  this  could  mean 
ten  hours  of  lost  time  per  month  or  600  hours  over  the 
life  of  the  system.    This  lost  time  is  estimated  to  be 
worth  $12,000. 

SOURCE:    U.S.,  Department  of  the  Navy,   Specification,   Selection,  and 
Acquisition  of  Automatic  Data  Processing  Equipment,   Secretary  of  the  Navy 

Instruction  10462.13,  Department  of  the  Navy,  Washington,  D.C.   2-390,  p.  B-15. 

Figure  2  — -  Statements  of  Value  -  Continued 


because  many  of  the  mandatory  and  desirable  features  relate  to  the  vendor's 
capability  to  complete  the  workload  satisfactorily. 

The  review  of  the  using  activities  workload  is  concerned  with  insuring  that  the 
expected  workload  Is  described  in  terms  of  the  expected  levels  of  workload  and  in 
terms  of  the  representative  programs  or  tasks  plus  the  related  extension  factors 
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applicable  to  the  workload  levels.    Each  of  these  factors,  workload  levels  and 
representative  programs,  will  be  discussed  individually. 

Exp e cted  Workload  Levels 

The  workload  to  be  processed  on  any  system  can  be  considered  as  consisting  of 
a  series  of  various  levels.    As  the  system  ages,  the  workload  usually  increases 
from  one  level  to  the  next.    However,  because  of  the  problems  associated  with 
projecting  future  workload  growth,  it  v/ould  be  unreasonable  to  expect  that  the  user 
could  be  100  per  cent  accurate  in  projecting  the  workload  for  the  life  of  the  system. 
Accordingly  the  user  is  directed  to  use  probabilities  to  predict  future  workloads  in 

o 

the  following  manner: 

1.         System  Life  -  the  user  prepares  a  chart  that  shows  the  number  of  years 
the  system,  will  be  in  existence. 


a.     SYSTEM  LIFE 


1st  Year 


2nd  Year 


3rd  Year 


4th  Year 


5th  Year 


Joslin,  "Competitive  Computer  Selection,"  p.   13. 
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2.         Projected  Growth  -  an  approximation  of  the  predicted  growth  in 
workload  is  superimposed  on  the  basic  "System  Life"  chart  a.  .    The  vertical  axis 
represents  workload  in  hours-per-month. 


b.    PROJECTED  GROWTH 


3.        Workload  Levels  -  the  user  constructs  a  workload  level  line  by  drawing 
a  horizontal  line  through  the  midpoint  of  the  projected  growth  line  for  each  year. 


"O 

c. 

WORKLOA] 
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1st  Year 

2nd  Year 

3rd  Year 

4th  Year 

5th  Year 

Level  5 
Level  4 
Level  3 
Level  2 
Level  1 


4.         Level  of  Probability  -  for  each  year  a  probability  factor  is  assigned  to 
each  workload  level.    The  total  probability  for  any  one  year  is  considered  to  be  100. 
In  illustration  d.  the  user  has  indicated  that  during  the  first  year  there  is  a  90  per 
cent  chance  that  the  workload  will  be  at  Level  1  and  a  10  per  cent  chance  it  will  be  at 


28 


Level  2.    Note  in  the  illustration  below  that  the  number  of  workload  levels  was 
increased  to  seven.    This  was  done  to  take  care  of  a  situation  where  there  is  a 
probability  the  workload  will  exceed  the  projected  growth  line  in  the  fourth  and  fifth 
years. 


d.    LEVEL  OF  PROBABILITY 
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Representative  Programs 


This  is  a  very  important  part  of  the  systems  specifications.    The  representative 
programs  are  the  vehicles  that  will  be  used  to  time  the  proposed  computer  systems. 
The  essence  of  this  approach  is  that  a  program  called  a  "representative  program"  is 
selected  to  represent  a  specific  portion  of  the  total  monthly  workload.    Thus,  a  group 
of  representative  programs  represents  the  total  monthly  workload.    These  programs 
which  become  benchmarks,  in  conjunction  with  the  extension  factors,  provide  the  tools 
to  project  the  monthly  running  time  for  the  systems  proposed. by  each  vendor. 

In  view  of  the  importance  of  this  factor  the  review  conducted  by  ADPESO  is 
particularly  critical.    The  reviewer  looks  to  insure  that  certain  facets  are  properly 
addressed.    For  example,  the  benchmarks  should  be  written  in  a  standard  higher 
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level  programming  language  such  as:    USASI  Fortran  or  USASI  COBOL.    The  vendor 
should  bo  able  to  run  all  programs  during  a  single  half-day  benchmark  demonstration. 

The  reviewer  has  to  insure  that  the  user  has  selected  a  truly  representative 
program.    This  means  that  the  benchmark  problem  must  be  representative  of  the 
following:" 

1.  Types  of  processing:  logic,  computation,  housekeeping,   etc. 

2.  Time  requirements:  compile,  execute,  tapes,  card  units,  printers,  etc. 

3.  Equipment  Requirements:  core,   input/output  channels,  tapes,   card  units, 
printers,  etc. 

4.  Order  of  problems  (priority). 

The  review  also  has  to  establish  that  the  user  has  properly  equated  each 
benchmark  to  a  portion  of  the  workload.    This  is  particularly  important  for  as 
previously  mentioned,  the  performance  demonstrated  on  the  benchmark  is  extra- 
polated to  define  the  total  monthly  workload. 

The  extension  factor,  a  number  derived  for  each  representative  program,   is 
the  next  vital  element  to  be  verified.    The  running  time  of  each  benchmark  is 
multiplied  by  this  factor  to  arrive  at  the  monthly  running  time  for  the  class  of  tasks 
represented  by  that  program.    An  extension  is  required  for  both  sequential  systems 
and  for  multi-processing  systems.    The  user  should  have  prepared  a  table  similar 
to  Figure  3/    For  each  representative  program  there  should  be  entries  in  the  table 
showing  the  total  monthly  time  consumed  by  each  program,  the  running  time  of  the 
representative  task  or  program  and  the  division  of  the  total  by  the  representative 


o 

Edward  O.  Joslin,  Computer  Selection  (Reading,  Massachusetts:  Addison- 

Wesley  Publishing  Company,    196S),  p.  77. 
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Workload 

Time 
(Hours  and  Hundredths) 

Extension 
Factors 

Task  Set 

Functions 

Monthly 

Task' 

Representative 

Task 

(Single  Run) 

Sort 

Total  thruput 

145.00 

0.45 

322 

B-1A 

Mag.  tape 

125.00 

0.25 

500 

Card  Reader 

115.00 

0.03 

3833 

Edit 

Total  thruput 

120.00 

0.75 

160 

E-4a 

Mag.  tape 

80.00 

0.G0 

ltlO 

Card  Reader 

20.00 

0.50 

4  0 

Printer 

100.00 

0.25 

400 

Update 

Total  thruput 

100.00 

0.16 

625 

D-5a 

Mag.  tape 

70.00 

0.10 

'700 

Card  Reader 

25.00 

0.05 

500 

Printer 

50.00 

0.10 

500 

Matrix 

Total  thruput 

90.00 

0.45 

200 

Inversion 

Card  Reader 

3.50  ' 

0.02 

175 

K-6a     ■ 

Mag.  drum 

24.00 

0.15 

160 

Printer 

1.50 

0.01 

150 

FORTRAN 

Total  thruput 

85.00 

0.17 

500 

Compile 

Mag.  tape 

78.00 

0.15 

520 

H-3a 

■  Card  Reader 

6.00 

0.02 

300 

Printer 

4.00 

0.01 

400 

COBOL 

Total  thruput 

40.00 

0.12 

333 

Compile 

Mag.  tape 

38.00 

0.11 

345 

.G-2 

Card  Reader 

4.00 

0.04 

100 

Printer 

3.00 

0.04 

75 

Tape  to 

Total  thruput 

300.00 

1.00 

300 

Print 

Mag.  tape 

300.00 

1.00 

300 

F-4 

Printer 

300.00 

1.00 

300 

TOTAL  MONTHLY 

TIME 880.00 

SOURCE:   U.S.  ,  Department  of  the  Navy,  Specification,  Selection,  and 
Acquisition  of  Automatic  Data  Processing  Equipment,   Secretary  of  the  Navy 
Instruction  1 04 62 .  1  3^9epariTmeni:  ofthe  Navy",  Washington,  D.C.  20390,  p.  B--4_ 
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program  and  this  becomes  the  extension  factor.     Expressed  another  way,  the 
extension  factor  is: 

X        -        Y 
X        -        ^ 


Y 


X 


Total  monthly  time  to  perform  total  task  set  on 
present  equipment  system 

Throughput  time  to  run  the  representative 
program 

(1)  on  present  equipment  (subscript  a) 

(2)  on  proposed  equipment  (subscript  b) 

Extension  factor  for  representative  task. 


Therefore: 


X(Zt))        -        Suppliers  time  for  total  task*0 
The  user  also  has  to  show  how  these  individual  representative  programs  can 
be  grouped  into  a  mix  of  programs  that  has  its  own  extension  factor,  which  is 
representative  of  the  total  workload.    The  purpose  of  this  mix  of  representative 
programs  and  related  factors  is  for  use  in  demonstrating  the  multiprogramming 
or  multiprocessing  capability.    The  mix  of  programs  as  modified  by  quantity  and 
provisions,   should  be  such  that  when  the  times  for  the  present  representative 
programs  are  multiplied  by  the  extension  factors  the  result  will  be  the  total  time 
which  should  closely  approximate  the  present  total  time.     Figure  4  demonstrates 
this  situation. 


10 


SecNavInst  10462.13,  p.  B-5. 
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Program 

Quantity 

Provisions 

B-la 

2 

Normal 

E-4a 

1 

Input  from  tape 

D-5a 

4 

Twice  Normal 
Twice  no  output 

K-6a 

1 

Normal 

H-3a 

3 

Normal 

G-2 

2 

Normal 

.    F-4 

2 

Normal 

Extension  Factor  for  Mix:     160* 

*This  extension  factor  for  the  mix  is  derived  by  examining  the  information  contained 
in  Figure  4  and  obtaining  the  lowest  practical  extension  factor  to  reduce  the  number 
of  problems  to  be  run  in  the  mix  while  retaining  the  required  representative  nature  of 
the  mix  of  problems,  which  in  this  case  is  160.    This  extension  factor  is  then 
divided  into  each  of  the  sequential  extension  factors  to  obtain  the  quantity  column. 
The  provisions  column  is  then  used  to  make  the  input/output  total  time  when  extended 
by  the  mix  extension  factor  equal  to  the  total  projected  input/output  time.    This  mix 
of  tasks  (15)  can  then  be  used  as  a  proper  demonstration  of  a  supplier's  multi- 

programmmg  or  multi-processing.    _____ 

SOURCE:   U.S.,  Department  of  the  Navy,  Specification,  Selection,  and 
Acquisition  of  Automatic  Data  Processing  Equipment,  Secretary  of  the  Navy 
Instru ction  10462.  13,  Department  of  the  Navy,  Washington,  D.C.  2-390,  p.  B- 6 . 


Figure  4  — —  Extension  Factors 


The  extension  factors  initially  calculated  are  for  the  first  year  of  the  systems 
life.    Accordingly  the  user  must  also  provide  extension  factors  applicable  to  the 
second,  third,  fourth,  and  fifth  year  of  the  systems  life. 

The  final  check  the  reviewTer  makes  is  to  insure  that  the  user  has  provided  a 
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description  of  each  of  the  benchmark  programs  and  copies  of  the  program  detail 
listing. 

If  the  user  has  properly  justified  the  mandatory  requirements,  evaluated  the 
desirable  features,  and  described  the  workload,  the  specifications  are  ready  for  the 
next  action.    The  specifications  are  forwarded  to  the  Source  Sek+Hxm  Evaluation 
Board  for  the  preparation  of  the  Request  for  Proposal  which  will  be  forwarded  to  the 
vendors,  and  the  Selection  Plan  which  is  held  in-house  for  official  use  only. 

Preparation  of  the  RFP  and  Selection  Plan 

Provided  the  specifications  were  properly  prepared,  the  preparation  of  the  RFP 
and  Selection  Plan  is  a  relatively  simple  task. 

The  RFP  consists  of  a  cover  letter,  the  systems  specifications,  less  the 
justification  of  mandatory  requirements  and  the  rationale  for  desirable  features,  and 
a  set  of  instructions  addressed  to  the  vendors.    These  instructions  tell  the  vendors 
how  to  prepare  their  proposals;  lists  some  mandatory  administrative  details  and 
defines  how  the  cost  for  the  proposals  should  be  computed  and  shown.    Before  this 
can  be  mailed  to  the  vendors  through,  the  Selection  Plan  has  to  be  developed. 

The  purpose  of  the  Selection  Plan  is  to  establish  a  detail  set  of  instructions  that 
will  govern  how  the  vendor  proposal  s  will  be  evaluated.     It  should  be  noted  at  this 
point  that  in  the  Navy's  system  it  is  the  desirable  features  requested  by  the  user  that 


Josiin,  "Competitive  Computer  Selection,  p.  22. 
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are  evaluated.    The  mandatory  requirements  are  not  evaluated  they  are  validated.  ^ 
The  desirable  features  and  die  values  assigned  to  each  are  carefully  analyzed 
Based  on  this  information  a  value  templet  is  constructed  for  each  feature.    These 
templets  show  how  the  total  value  for  each  item  will  be  apportioned  as  various 
quantities  or  qualities  of  the  item  are  proposed.    In  actual  practice  these  value 
statements  are  prepared  when  the  RFP  is  being  prepared.    The  purpose  of  tiiis  is  to 
insure  agreement  between  the  RFP  and  the  Selection  Plan. 

Another  examination  of  the  mandatory  requirements  is  also  conducted  at  this 
time.    The  reason  for  this  second  review  of  these  requirements  is  to  determine  if 
any  items  initially  classified  as  mandatory  can  be  re- classified  as  desirable. 
Experience  has  shown  that  some  mandatory  items  are  really  desirable  in  the  sense 
that  dollars  can  sometimes  make  up  for  a  slight  deficiency  in  a  mandatory 
requirement. 

There  are  four  parts  in  a  value  statement: 

1.  A  Statement  of  the  Desirable  Feature  -  provides  the  exact  and  entire 
statement  of  the  desirable  feature  as  it  appears  in  the  RFP. 

2.  Pvationale  -  This  section  shows  the  derivation  of  the  worth  or  value  of 
the  feature.    The  worth  is  defined  by  considering  the  following: 

a.  The  cost  of  doing  without  the  capability; 

b.  The  cost  of  satisfying  the  requirement  by  using  in-house  resources; 


12SecNavlnst  10462.13,  p.  B-ll. 

-^^Joslin,  "Competitive  Computer  Selection,"  p.  23. 
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c.  The  cost  of  having  the  capability  provided  by  some  contractor  other 
than  the  one  bidding  on  the  system  (must  always  be  preceded  by  a.  or  b. ); 

d.  A  cost  less  than  any  of  the  above  which  would  reflect  what  the 
activity  would  apy  to  get  this  capability  (must  be  preceded  by  a. ,  bo ,  ore);  and 

e.  The  worth  of  each  feature  must  be  calculated  for  each  year  of  the 
systems  life. 

The  lowest  worth  derived  above  becomes  the  value  of  the  feature. 

3.  Templet  -  tells  how  the  worth  or  value  derived  is  to  be  distributed. 

4.  Valuation  -  is  actually  done  on  a  separate  standard  form  which  is 
attached  to  the  value  statement  sheet.  (A  sample  value  statement  is  shown  in 
Figure  5.) 

The  Selection  Plan  also  defines  how  the  costing  of  the  vendors  proposal  will  be 
accomplished.    It  is  at  this  point  that  many  of  the  previously  mentioned  elements 
start  coming  together,  e.g.  ,  the  workload  levels,  extension  factors,  time  required 
to  execute  a  benchmark,  etc.    The  cost  of  money  is  introduced  at  this  point. 
There  is  a  requirement  to  perform  an  economic  analysis  of  the  cost.  ^    Currently 
the  discount  rate  in  use  is  10  per  cent. 

An  example  of  the  formula  used  to  define  the  cost  of  a  system  is  shown 
below.    The  basis  for  this  example  is  the  workload  levels  and  probabilities 


U.S.  ,  Department  of  the  Navy,  "Economic  Analysis  of  Proposed  Department 
of  the  Navy  Investments,"  Comptroller  of  the  Navy  Instruction  7000.  28  (  April  19, 
1967)  Department  of  the  Navy,  Washington,  D.C.  20390. 
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VALUE  STATEMENT 


DESIRABLE  FEATURE;   Capability  to  expand  at  year  2,  the  systems  at  the 
three  activities  to  include  time  sharing  systems  for  program  testing 
purposes. 

RATIONALE:   "Time  Sharing  Versus  Batch  Processing:   The  Experimental 
Evidence"  by  H.  Sockman,  10  October  1967,  AD  661-665,  Defense  Documen- 
tation Center  publication,  reports  that  a  25%  reduction  in  program 
development  time  lias  been  shown  in  experimental  studies  by  using  time- 
sharing teehniques  instead  of  bate!)  processing  techniques  in  testing. 
Time-sharing  caused  a  40%  increase  in  computer  test  time.   No  signi- 
ficant differences  were  found  in  program  size  or  run  time.   On  this 
basis,  the  following  cost  reduction  would  he  gained  from  time-sharing: 

25%  of  -all  programmers'  yearly  salary  -  40%  X  hourly 
computer  rental  rate  X  hours  test  time,  per  week  X  52 
weeks  per  year  -  cost  reduction/base. 

Taking  number  of  programmers  at  each  branch  =  20 

Programmer  yearly  salary  -  $20,000 

Hourly  computer  rental  rate  =  $100 

Average  hours  test  time  per  week  -  20 

Then  (.25  X  $20,000  X  20)  -  (.40  X  $100  X  20  X  52)  -  cost  reduction  per 
year  for  one  activity. 

$100,000  -  $41,600  -  $58,400  yearly  savings  per  activity 

X3  activities 
$175,200  total  per  year 

X4  years 

$700,800  total  savings 

TEMPLET :   If  proposed,  a  value  of  $58,400  would  be  given  for  each  activity 
per  year  for  each  year  after  the  first  year. 

SOURCE:    E.O.  Joslin,  "Competitive  Computer  Selection  Within  the 
Department  of  the  Navy,"  unpublished  paper  presented  to  the  Diebold  Group  Inc.  , 
October  1969. 


Figure  5  —  Sample  Value  Statement 
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shown  on  page  28.      In  this  case,    there  is  a  90  per  cent  chance  of  the  workload 
being  at  Level  1  for  the  first  twelve  months,  and  there  is  a  5  per  cent  chance  this 
will  be  the  case  in  the  second  twelve  months.    This  same  type  of  analysis  is 
performed  for  all  of  the  other  workload  levels.    Based  on  this  data  the  costing- 
formula  is  then  developed.    The  formula  for  the  first  world oad  level  would  be: 
M^wl  n  =  monthly  cost  workload  level  n 

Y  =  number  of  months 

P  =  probability  of  operating  at     , 

D  =  discount  rate 

YMCwl  n    •    P    •    D  +   YMCwl  n    "    P    •    D 
12MCwl  x   x   0.90   x    0.91    +    12MCwl  1    x   0.05    =    10. 33MCwl  y 

By  performing  the  same  mathematical  operation  to  the  other  workload  levels,  the 
present  worth  of  total  expected  lease  payments  for  each  configuration  can  be 
expressed  as: 
10.33MCwl  1   +    10.00MCwl2   +    8.56MCwl  3  +   7-75MCwM   +   6. 89MCwl  g   + 

i-^^w^   +   °-37MGwl7   -15 
The  Selection  Plan  is  now  complete.    The  SSEB  presents  the  RFP  and  the  plan 
to  the  Source  Selection  Advisory  Council  for  approval.    When  this  review  is  completed 
and  approval  is  granted,  the  RFP  is  forwarded  to  the  suppliers. 


^U.S.  ,  Department  of  Defense  Computer  Institute,.  Report  of  the  Fourth 
Computer  Selection  Techniques  Seminar  (Washington,  D.  C. :  February  18-19,   1969), 
p.  8. 
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Vendor  Liaison  and  Validation 

The  amount  of  effort  that  has  to  be  devoted  to  this  function  is  inversely  related 
to  the  degree  of  excellence  in  the  RFP.    The  period  of  time  is  normally  four  months. 
Essentially  all  that  is  required  of  the  Navy  personnel  is  to  be  available  to  answer 
vendor  questions,  and  to  be  ready  to  witness  the  running  of  the  benchmark  programs 
when  the  vendors  are  ready  to  demonstrate  them. 

There  is  one  cardinal  rule  in  vendor  liaison:   whatever  is  told  or  applies  to 
one  vendor  must  be  told  to  all  vendors,  and  in  writing.    This  rule  is  considered 
essential*-*- it  is  often  difficult  to  differentiate  between  liaison  and  negotiation.     It 
is  also  necessary  in  order  to  avoid  later  difficulties  which  can  arise  when  all 
bidders  are  not  equally  informed. 

Proposal  Evaluation 

Evaluation  of  the  vendors  proposals  is  not  a  simple  task  if,  however,  the  RFP 
and  Selection  Plan  is  properly  prepared  the  degree  of  difficulty  will  be  reduced 
considerably.    The  actual  steps  included  in  the  evaluation  process  are: 16 

1.  Reading  and  understanding  all  proposals. 

2.  Verifying  that  all  mandatory  conditions  are  satisfied.  If  a  proposal  does  not 
satisfy  all  manditory  conditions  it  is  ruled  not  responsive  and  it  is  no  longer 
considered. 

3.  Determining  the  degree  to  which  the  requested  desirable  features  have  been 
provided. 


^"Joslin,  "Competitive  Computer  Selection,"  p.   29. 
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4.  Matching  the  response  to  the  desirable  features  with  the  value  statements. 

5.  Costing  the  vendors  proposal. 

The  verification  of  vendor  claims  is  often  difficult.    To  assist  in  this,  the  RFP 
requests  the  vendors  to  reference  the  technical  manuals.     If  the  SSEB  cannot  verify 
a  claim  by  using  the  technical  manuals  or  other  references,  the  capability  in  question 
is  regarded  as  a  vendor's  promise  and  it  is  incorporated  as  a  term  and  condition  in 
any  resulting  contract. 

Determining  the  degree  to  which  a  desirable  feature  is  satisfied  and  matching 
this  with  the  value  statement  is  also  a  difficult  task.  "It  can  result  in  many  bruised 
egos,  but  normally  it  can  be  worked  out  to  the  satisfaction  of  all.  "™ 

The  pricing  or  costing  of  the  life  of  a  system  is  an  involved  process.    The  total 
cost  is  dependent  on  many  elements:  the  number  of  years  the  system  will  be  used, 
the  number  of  hours  the  equipment  will  be  used,  the  cost  of  the  desirable  features 
provided  by  the  vendor,  the  cost  of  satisfying  a  desirable  feature  by  an  alternate 
means  if  the  vendor  does  not  provide  the  item,  the  cost  of  vendor  support,  etc. 

Another  complicating  factor  is  that  each  of  these  costs  have  to  be  considered 
under  several  different  procurement  possibilities:  lease,  lease  with  purchase  option, 
purchase,  and  any  special  procurement  plan  which  may  be  worked  out  with  vendors.-1 9 
In  addition  and  as  previously  mentioned,  an  economic  analysis  using  the  present  value 
technique  must  also  be  included  in  the  calculations. 


17Ibid.  18Ibid. 

19Ibid.,  p.  31.  20NavComptIn st  7000.28. 
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Negotiation  and  Contract  Award 

The  SSEB  review  of  the  vendor  proposals,   is  attended  by  a  continuous  search 
for  items  which  might  be  negotiable.    An  example  is  an  item  offered  by  one  vendor 
at  no  cost  might  be  obtainable  free  from  the  other  vendors  if  the  item  was  subjected 
to  negotiation.    Usually  such  items  as  training,  maintenance,  and  some  software 
packages  fall  into  this  category. 

After  the  SSEB  has  defined  which  proposals  are  responsive  to  the  requirements 
of  the  RFP,  letters  are  prepared  and  sent  to  the  vendors.    The  purpose  of  this  letter 
is  to  set  up  the  time  and  place  for  negotiation  sessions  with  the  responsive  bidders, 
hi  addition  the  letter  tells  the  vendor  what  items  the  Navy  wishes  to  negotiate  with 
the  vendors. 

At  the  negotiation  session,  the  vendors  and  Navy  personnel  discuss  the  items 
in  question.    The  vendor  is  told  when  to  submit  the  final  proposals.    These  proposals, 
however,  must  be  based  on  the  original  proposal  plus  information  covered  in  the 
negotiation  sessions.     Each  of  the  contractors  is  encouraged  to  improve  his  proposal, 
and  they  are  reminded  to  submit  them  by  the  scheduled  date  because  late  proposals 
might  not  be  considered.    The  session  is  then  closed,  and  a  memorandum  of  the 
meeting  is  prepared  and  becomes  a  part  of  the  contract  pre- award  file.  21 

When  the  proposal  changes  arc  received,  the  SSEB  again  reviews  the  proposals 


21Joslin,  "Competitive  Computer  Selection,"  p.   32. 
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and  adjusts  the  cost  data  based  on  the  changes  received.    The  evaluation  then 
determines  the  proposal  representing  the  lowest  overall  cost  to  the  Department 
of  the  Navy.    The  selected  proposal  is  nominated  to  the  SSAC  and  to  the  Special 
Assistant  to  the  Secretary  of  the  Navy  (SASN). 

When  the  SASN  approves  the  selection,  three  copies  of  the  contract  are  sent  to 
the  vendor  for  signature.     If  the  supplier  disagrees  with  some  aspect  of  the  contract, 
the  point  is  discussed  and  resolved.    The  vendor's  representative  and  the  government 
contracting  officer  then  sign  the  contract. 

Occasionally  there  is  one  other  step.     If  the  award  is  for  more  than  $  1,  000,000, 
ADPESO  does  not  announce  the  award  for  at  least  twenty  working  hours  after  notifying 
the  Office  of  Legislative  Affairs.    The  purpose  of  this  hold  is  to  allow  the  appropriate 
Congressman  to  be  told  of  the  award.  22 

There  are  several  other  actions  that  are  initiated  at  this  time.    An  announce- 
ment of  the  award  is  sent  to  the  Commerce  Business  Daily  for  public  announcement 
via  that  media.     Letters  are  sent  to  the  unsuccessful  bidders  indicating  that  their 
proposal  was  not  selected  however  their  future  interest  is  solicited.    This  letter  also 
provides  the  unsuccessful  bidder  an  opportunity  for  a  post- approval  meeting  with 
ADPESO.     If  any  unsuccessful  bidder  desires  such  a  meeting  a  written  request  has  to 
be  sent  to  ADPESO  within  ten  days.    There  are  several  guidelines  established 


22Ibid.  ,  p.   33. 
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governing  the  conduct  of  these  post- approval  meetings: 

1.  No  informal  sessions  with  respect  to  approved  equipment  selection  shall 
be  held  with  any  unsuccessful  supplier,  nor  shall  any  oral  information  be 
furnished  on  the  subject. 

2.  All  questions  from  an  unsuccessful  supplier  shall  be  provided  to  ADPESO 
in  advance  and  in  writing,  and  should  be  as  specific  as  possible.    The  DON 
reserves  the  right  to  reject  irrelevant  or  immaterial  questions.     Conversely, 
related  and  relevant  questions  which  are  obviously  and  naturally  evoked  by 
responses  to  previously  submitted  questions  maybe  entertained  within  reason. 
In  this  regard,  however,  the  normal  provisions  apply  with  respect  to  security 
sensitive  information  and  proprietary  information. 

3.  The  judgement^exercised  in  the  application  of  selection  criteria  to 
particular  proposals  is  properly  a  matter  wholly  within  DON  purview. 

4.  Post- approval  meetings  shall  be  held  as  soon  after  they  are  requested  as 
practicable.  ^ 


23SccNavInst  10462.13,  p.   Ill- 9. 


CHATTER  IV 
ALTERNATIVE  SELECTION  SYSTEMS 

To  assist  prospective  users,  today  there  are  a  number  of  different  methods 
used  in  determining  which  computer  system  best  satisfies  the  data  processing- 
requirements  of  the  organization.     Some  organizations  select  their  computers  by 
soliciting  competitive  bids  while  others  select  a  system  without  seeking  any  vendor 
competition.    When  competitive  bids  are  requested  the  criteria  used  to  determine 
the  best  system  ranges  from  objective  standards  to  subjective  judgement.    The 
purpose  of  this  chapter  is  to  review  the  various  current  practices  and  explore  the 
details  of  some  of  the  systems  used. 

Survey  of  Computer  Selection  Practices 

Dr.  Norman  F.  Schneidewind  of  the  Advanced  Systems  Division,  Systems 
Development  Corporation,  conducted  a  survey  for  Datamation  and  the  results  were 
subsequently  published  in  the  magazine.  1    The  data  was  obtained  by  sending 


'■Norman  F.  Schneidewind,  "The  Practice  of  Computer  Selection,"  Datamation, 
13.2  (February  1967),  pp.   22-25. 
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questionnaires  to  four  types  of  computer  users.    The  types  of  users  and  the  number 
responses  from  each  were: 

1.  Commercial:    Non-government,  non-aerospace,  non-university  37 

2.  Government:    Civilian,  military,  federal,   state,  local  20 

3.  Aerospace:    Aircraft,  missle,   space  8 

4.  University  4 


Total  responses  69 

The  survey  questionnaire  was  directed  toward  four  aspects  of  the  computer 
selection  process.    The  areas  covered  and  the  findings  of  the  survey  are  as  follows. 

Competition 

The  competition  section  of  the  questionnaire  was  designed  to  determine  whether 
the  selection  process  takes  place  on  a  competitive  or  sole  source  procurement  basis. 
The  responses  indicated  that  between  77  and  92  per  cent  of  all  users  utilize 
competitive  selection  systems.    The  degree  to  which  the  various  users  rely  on  the 

n 

competitive  process  is  shown  below. 


Government 

94.4% 

Non-government 

80.9% 

Commercial 
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Aerospace 
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75.0% 

Total  All  Users 

84.6% 

Dr.   Schncidwind  determined  that  in  general  it  was  the  user  with  a  small  installation 
(average  monthly  rental  of  $16,800)  that  used  single  source  procurement,  while  the 


2  Ibid. 
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management  of  larger  installations  (average  monthly  rental  of  $62,  300)  select  their 
equipment  based  on  a  competitive  bidding  system. 

Methods 

The  purpose  of  the  methods  section  of  the  questionnaire  was  to  determine  which 
of  the  following  selection  methods  are  used.    The  methods  are  listed  with  the  one 
considered  by  Dr.   Schneidewind  as  the  most  objective  first  while  the  last  was 
considered  to  be  the  least  objective: 

.  Computer  Simulation 

.   Mathematical  Modeling 

.  Program  and  Execute  Test  Problems 

.  Evaluation  of  Benchmark  Problems 

.  Published  Hardware  and  Software  Evaluation  Reports 

It  was  found  that  larger  installations  use  more  objective  methods  than  small 

3 

installations.    The  specific  findings  are  given  below. 
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3Ibid. 
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Selection  Criteria 

The  respondents  were  asked  to  rank  the  following  computer  selection  criteria 

in  order  of  importance: 

Hardware  performance 

Software  performance 

Support  provided  by  manufacturer 

Availability  of  application  programs 

Compatibility  with  present  hardware  and  software 

Delivery  date 

Potential  for  growth  (modularity) 

Cost 

The  user  was  asked  to  rank  these  items  by  assigning  the  number  one  to  the  most 

important,  the  number  two  to  the  next  most  important,  etc.     According  to 

Dr.  Schneidewind  the  most  objective  criteria  are  hardware  and  software  performance, 

while  the  lea  st  objective  are  ava  liability  of  application  programs  and  manufacturer  support. 

The  results  indicate  that  all  users  place  more  emphasis  on  the  objective 

(hardware  and  software)  criteria  than  on  the  subjective  criteria.    Dr.  Schneidewind 

considered  this  to  be  the  most  significant  finding  of  the  survey.     In  addition  the  results 

indicate  that  commercial  users  rated  cost  as  a  less  important  criteria  than  did  the 

4 
other  users.    The  average  rankings  are  shown  below. 

Commercial    Government    Aerospace    University 
Hardware 
Software 

Manufacturer  Support 
Availability  of  Programs 
Compatability 
Delivery 
Growth 
Cost 


3.11 

2.15 

1.57 

2.50 

2.73 

2.45 

3.68 

1.75 

3.92 

4.32 

5.29 

3.50 

6.57 

7.30 

7.  14 

6.75 

4.35 

4.85 

3.86 

6.00 

6.27 

6.45 

6.43 

7.25 

4.68 

4.25 

5.57 

4.50 

4.38 

4.25 

2.43 

3.  75 

4  Ibid. 
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Outside  Assistance 

The  intent  of  this  section  of  the  questionnaire  was  to  determine  the  extent  to 
which  users  depend  on  outside  assistance  in  the  selection  process  and  when  they  did 
what  type  of  consultant  was  used.     Specifically  the  respondents  were  asked  to  indicate 
which  of  the  following  types  of  outside  assistance  were  used: 

Independent  Consultant 

Accounting  Firm 

Management  Consultant 

Technical  Firm 

Computer  Manufacturer 
The  findings  indicate  that  the  number  organizations  using  outside  assistance  is 
small,  and  the  organizations  that  do  use  consultants  generally  have  small  data 
processing  installations.    When  outside  assistance  is  used  the  computer  manufacturers 
are  relied  on  more  heavily  man  the  others. 

Based  on  the  analysis  of  the  data  accumulated  in  this  survey,  Dr.   Schneidewind 
developed  what  he  considered  to  be  a  typical  computer  selection  process.     In  general, 
computer  selection  is  competitive,  objective  criteria  is  stressed,  and  the  selection  is 
performed  in-house  rather  than  by  an  outside  consultant. 
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Review  of  Specific  Alternative  Systems 

The  process  of  selecting  a  computer  from  among  several  alternative  proposals 

logically  breaks  down  into  two  major  parts.    The  first  part  consists  of  the  validation 

of  the  vendor's  claims.     The  validation  process  is  concerned  with  verifying  the 

capabilities  of  each  system  proposed  and  the  estimated  system  times  required  by 

each  configuration  to  complete  the  workload.    This  is  a  very  important  part  of  the 

selection  process.    The  user  must  be  aware  of  the  old  saying  "buyer  beware. "    While 

the  vendors  will  probably  not  perpetrate  a  falsehood  the  buyer  must  understand  that 

the  proposal  is  a  selling  device  and  certainly  each  vendor  wants  to  look  better  than 

all  others.     In  order  to  look  best,  the  vendors  will  interpret  the  specifications  in  a 

way  that  is  favorable  to  their  equipment.    The  amount  of  interpretation  is  related  to 

the  way  the  requirements  are  set  forth  in  the  specifications  of  the  RFP.     If  the 

specifications  are  loosely  worded  then  the  suppliers  are  put  into  a  situation  that 

requires  a  great  deal  interpretation  of  the  specifications  to  make  a  proposal.    An 

example  of  the  types  of  problems  that  can  arise  if  the  proposals  are  not  thoroughly 

validated  against  the  organizations  requirements  is  the  situation  at  United  Airlines 

and  Trans  World  Airlines: 

Two  of  the  largest  commercial  computer  systems  ever  ordered  —  for 
reservations  at  United  Airlines  and  Trans  World  Airlines — are  not  working  out. 
United  has  decided  to  drop  its  Univac  system,   estimated  to  be  valued  at  more  than 
$30  million,  while  TWA  reportedly  is  close  to  canceling  an  equally  large  system 
ordered  from  Burroughs  Corp.     According  to  United,   the  Univac  system  was  more 
than  a  year  late  and  still  inoperative.     Industry  sources  report  that  delivery 
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problems  in  hardware  and  misunderstandings  over  programming  tripped  up 
Burroughs  and  TWA.    Bodi  airlines  will  continue  to  use  their  existing  computers 
until  they  select  suppliers  for  a  third- generation  system.  5 

The  second  half  of  die  selection  process  is  the  evaluation  of  the  proposals  that 
satisfy  all  of  the  basic  data  processing  requirements.    The  purpose  of  the  evaluation 
is  to  differentiate  among  the  various  proposals  and  find  the  one  that  best  satisfies  all 
the  selection  criteria. 

There  are  several  methods,  besides  the  ones  used  by  the  U.  S.  Navy,  that  can 
be  used  in  the  validation  and  evaluation  of  system  proposals.    The  rest  of  this  chapter 
purports  to  examine  some  of  the  alternative  validation  and  evaluation  systems. 

Alternative  Validation  Methods 

Two  areas  of  each  proposal  must  be  validated:    system  capabilities  and  system 
timing. 

Capabilities 

The  user  can  verify  system  capabilities  in  several  ways.     First,  all  the 
manufacturers  technical  literature  can  be  read.    The  next  step  is  to  compare  this 
data  with  the  vendors  proposal.    An  alternate  way  would  prepare  a  detailed 


^"Airline  Computer  Problems,"  Business  Week,  February  7,   1970,  p.   34 
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questionnaire  to  be  included  in  the  RFP  and  request  the  suppliers  to  provide  the 
requested  data.    The  vendors  proposal  is  then  compared  with  this  data.    A  third  way 
of  obtaining  this  information  is  to  study  the  Standard  EDP  Reports  prepared  by  the 
Au  e  rb  a  cl  1  C  o  rp  o  ra  t  ion . 

Timing 

The  validation  of  system  timing  is  a  most  important  task.     In  addition,  it  is  a. 
difficult  task  to  accomplish,  particularly  with  a  high  degree  of  accuracy.    The 
importance  of  this  operation  however,   can  not  be  overemphasized.    The  timing- 
estimates  derived  will  serve  at  least  two  vital  functions  in  the  selection  process. 
First  this  information  will  be  used  to  determine  if  the  proposed  configuration  can 
perform  the  stated  workload.     Second,  the  timing  data  will  provide  the  foundation  for 
costing  the  proposal. 

There  are  several  ways  to  verify  the  timing  of  computer  systems:   benchmarks, 
computer  simulation,  and  detail  hand  timing.    The  use  of  benchmarks  was  described 
in  Chapter  III. 

Simulation 

Simulation  is  a  technique  in  which  a  model  of  the  actual  system  is  described  and 
constructed  and  testing  of  the  real  system  is  accomplished  by  exercising  the  model. 


Edward  O.  Joslin,  Computer  Selection  (Reading,  Massachusetts:  Addi son- 
Wesley  Publishing  Company,   Inc.  ,    1968),  p.   67. 
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The  function  then  of  a  simulation  model  is  to  show  the  results  created  when  certain 

7 
things  happen  under  the  control  of  a  set  of  decision  rules.      The  limitations  and  the 

advantages  of  simulation  are  stated  by  Teichroew  and  Lubin: 

Simulation  problems  are  characterized  by  being  mathematically  intractable  and 
having  resisted  solution  by  analytic  methods.  The  problems  usually  involve  many 
variables,  many  parameters,  functions  which  are  not  well  behaved  mathematically 
and  random  variables.  Thus,  simulation  is  a  technique  of  last  resort.  Yet,  much 
effort  is  now  devoted  to  "computer  simulation"  because  it  is  a  technique  that  gives 
answers  hi  spite  of  its  difficulties,  costs  and  time  required.  8 

There  are  several  computer- simulation  techniques  in  use  or  under  development; 
however,  the  system  most  widely  known  and  used  is  one  called  £>CERT  (Systems  and 
Computers  Evaluation  Review  Technique).       SCERT  has  the  capability  to  simulate 
over  100  different  computer  systems  manufactured  by  the  thirteen  different  companies 
companies.     A  complete  list  is  shown  in  Appendix  G.     SCERT  divides  into  four 
major  components:    definition  languages,  a  factor  library,   simulation  programs, 
and  output  reports. 

The  definition  languages  are  used  to  define  the  applications  systems  to  be 
simulated  and  the  hardware/software  complexes  to  be  simulated.    Several 


'John  E.   Cremeans,  "The  Trend  in  Simulation,"  Computers  and  Automation, 
January,    1968,  p.  46. 

c 

Daniel  Teichroeu  and  John  F.  Lubin,  "Computer  Simulation-Decision  of  the 
Technique  and  Comparison  of  Languages,"  Communications  of  the  ACM,  IX  (October 
1966),  p.   724. 

^Joslin,   Computer  Selection,  p.  86. 

A  Technical  Description  of  SCERT  (Rockville,   Maryland:  Comress,   Inc.), 
pp.    17-49.  ~ 
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language  section  components  facilitate  this.    The  data  definition  division  defines 
the  files  and  reports  of  the  applications  system.    The  procedures  division  defines 
each  application  program.     The  configuration  division  defines  each  equipment 
configuration  to  be  simulated,  including  model  number  and  quantity.    The  environment- 
division  defines  the  staff  that  support  the  installation  plus  salary  level. 

The  factor  library  contains  the  characteristics  of  the  hardware  arid  software 
items — characteristics  such  as  cost,  performance  and  technical  specifications.    This 
data  is  obtained  by  Comress  from  the  manufacturers. 

The  simulation  programs  perform  the  processing  necessary  to  accept  the  input 
definition  data  and  create  the  output  reports.  '  The  programs  are  used  in  five  phases 
which  will  be  discussed  later. 

The  output  reports  consist  of  fifteen  different  reports.    All  reports  are  not 
created  for  each  evaluation;  for  instance,  the  real-time  reports  would  not  be 
produced  when  a  batch  processing  system  is  being  simulated. 

The  entire  SCERT  operation  is  shown  as  Figure  6.    As  can  be  seen  there  are 
five  functional  phases. 

o  Phase  1  -  The  Introduction  of  Processing  Requirements 

The  first  input  to  SCERT  is  a  series  of  definitions  outlining  the  workloads  and 
computer  processing  requirements  of  the  system  to  be  simulated.     Specifically  the 
information  entered  is  concerned  with  three  areas. 


Unless  otherwise  footnoted,   this  and  future  information  relating  to  SCERT  was 
was  generally  obtained  from  A  Technical  Description  of  SCERT  (Rockvillc,   Maryland: 
Comress,  Inc.). 
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SOURCE:    A  Technical  Description  of  SCERT  (RockviUc,  Maryland:  Comress, 
Inc.),  p.    16. 


Figure  6  —  The  SCERT  Program 
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Systems  Environment  information  relates  to  the  environment  in  which  the 
computer  and  data  processing  system  will  operate.    The  detail  data  elements  are: 

Cost  Data  -  Specifies  the  costs  of  operation  such  as  programmer  and 
operator  salaries,   and  systems  life  expectancy. 

Programmer  Experience  Profile  -  the  user  indicates  the  experience  and 
qualifications  of  the  programming  personnel. 

Definition  Percentiles  -  the  user  indicates  the  percentage  of  the  total 
system  requirements  not  defined  for  analysis  by  SCERT.    The  simulator  uses  these 
to  project  the  total. 

Real  Time  -  the  parameters  concerning  real  time  processing,   if  any,  are 
entered. 

File  Definition  defines  each  file  in  the  data  system.     Each  file  is  given  a  unique 
number;  the  size  of  the  file  is  specified  by  indicating  the  number  of  records;  the 
number  of  characters  and  fields  per  record  and  the  type  of  file  is  indicated. 

Systems  Definition  is  the  common  denominator  for  defining  the  processing- 
requirements  in  the  individual  computer  run  or  random  event.     Each  run  definition 
consists  of  three  distinct  levels: 

Pain  identification,   frequency,   priority,   and  prerequisites.     Each  run  is 
defined  by  a  unique  number  and  the  frequency  is  indicated  as  daily,  weekly,   monthly, 
quarterly,   or  yearly. 

File  identification  and  through-  put  parameters.   Each  file  is  defined  by  the 
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unique  number  used  in  the  file  definition  above.    Any  special  considerations 
concerning  the  way  the  data  is  to  be  handled  is  indicated. 

The  internal  activity  section  provides  the  simulator  with  the  internal 
processing  applicable  to  the  input  and  output  files. 
Phase  1  has  three  primary  functions: 

.     Accept  the  input  definitions  indicated  above; 
.    Build  a  mathematical  model  of  each  computer  run;  and 
.    Validation  of  the  model  and  output  diagnostic  data  if  any  errors  or 
inconsistencies  are  found. 
o  Phase  2  -  The  Introduction  of  Hardware/Software  To  Be  Simulated 

In  Phase  2  the  specific  hardware  and  software  configuration  being  appraised  is 
introduced.    This  is  a  very  specific  and  detailed  definition  of  the  hardware.    The 
model  numbers  of  every  component,  all  special  features,  adaptors,  number  of 
channels,  etc.  to  be  used  have  to  be  provided.     In  addition,  the  details  applicable  to 
the  software  package  have  to  be  indicated. 
The  functions  of  Phase  2  are: 

.    To  accept  the  hardware/ software  definitions; 

.    To  build  a  mathematical  model  representing  hardware/ software;  and 
.    To  validate  the  compatibility  of  the  models  built  in  Phase  1  with 
Phase  2  models.     If  discrepancies  are  found,  a  diagnostic  report  is 
created. 
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o  Phase  3  -  Presimulation  Algorithms 

In  this  phase  the  models  built  in  Phase  1  are  passed  against:  the  models  of 
Phase  2.     Daring  this  processing  a  series  of  calculations  are  performed  which  hi 
effect  structure  and  parameter  the  non-hardware  oriented  models  to  the  performance 
abilities  and  capabilities  of  die  hardware. 

During  Phase  3  the  following  events  for  each  program  occur. 

The  internal  processing  time  and  memory  requirements  are  calculated. 
All  files  are  assigned  to  I/O  devices  and  channels. 
All  files  are  structured  to  meet  the  hardware  parameters. 
The  thru-put  time  for  all  I/O  and  memory  functions  is  determined. 
The  timing  of  the  software  packages  is  determined. 
The  pre-  and  post-run  times  for  factors  such  as  setup-time,  error- 
correction,  etc.  are  calculated. 
o  Phase  4  -  Simulation 

In  this  phase  the  actual  simulation  of  the  system  takes  place.  All  of  the  work 
performed  in  the  first  three  phases  was  in  preparation  of  the  data  for  input  to  Phase 
4.    There  are  three  distinct  stages: 

Stage  1  -  Each  program  run  or  random  event  is  expanded  into  its  maximum 
number  of  unique  thru-pat  iterations.    The  processing  simulates  the  flow  of  each  of 
these  thru-put  iterations  through  the  computer  configurations.    The  results  are  the 
net  thru-put  timing. 
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Stage  2  -  This  stage  is  entered  only  if  the  defined  processing  includes 
random- o  ecu  ring  events.    This  stage  constructs  models  based  on  probability  theory 
and  distributes  the  occurrences  of  events  in  a  probabilistic  manner.    These  models 
are  then  simulated  and  potential  queueing  points  are  determined,  and  processing 
delays  caused  by  the  queues  are  derived. 

Stage  3  -  This  stage  is  entered  whenever  the  hardware/ software  complex 
is  capable  of  multi-programming  or  multi-processing.     In  the  first  two  stages  the  net 
elapsed  time  and  capacity  requirements  were  developed.     It  is  the  function  of  Stage  3 
to  schedule  the  concurrent  processing  of  these  events.    When  this  is  completed,  the 
results  stored  in  the  mathematical  models  is  ready  to  be  printed  out. 

o  Phase  5  -  Production  of  Output  Reports 

SCERT  can  provide  the  user  with  fifteen  standard  reports.     If  requested  the 

simulator  can  be  modified  to  provide  nonstandard  or  special  purpose  reports.    The 

reports  that  are  of  particular  interest  in  validation  are  given  below.    A  complete  list 

of  the  standard  SCERT  reports  is  shown  in  Appendix  H. 

Computer  Complement  Report — Portrays  the  exact  configuration  that  was  simulated 
and  provides  certain  basic  cost  data  about  the  configuration. 

Central  Processor  Utilization— Summarizes  for  each  scheduled  run  the  projected 
running  time  and  horizontal  memory  utilization. 

Cost  Summary — Primary  cost  output  which  relates  the  projected  utilization  of  the 
computer  to  lease,  purchase,   and  maintenance  cost  considerations. 

Detail  Analysis — Produced  for  each  scheduled  run  or  real-time  event  simulated 
and  serves  as  backup  for  all  other  reports.     It  precisely  portrays  the  utilization, 
timing,   and  memory  requirements  derived  by  the  simulation  and  presimulation 
algorithms  for  all  components  making  up  the  run. 
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Hand  Timing 

The  essence  of  hand  timing  is  to  analyze  each  and  every  function  that  the 
computer  system  will  be  required  to  perform  during  the  running  of  a  program.    The 
hardware  engineering  time  for  each  operation  is  determined  and  all  of  these  times 
are  added  together  to  derive  the  program  running  time. 

This  method  of  timing  requires  the  user  to  have  a  detail  knowledge  and 
understanding  of  the  programs  that  are  going  to  be  timed,     hi  addition,  a 
standardized  method  of  gathering  the  required  data  should  be  used.    This  is  necessary 

hi  order  to  obtain  consistency  between  vendors  and  limit  the  number  of  approximations 

1  9 
in  the  timing  data.     Some  suggested  timing  tables  are  proposed  by  Joslin.        There 

are  several  data  elements  that  should  be  in  a  detail  time  table: 

Program  Number  and  Frequency 

Input/Output  Requirements 

Central  Processor  Memory  Requirements 

Non-productive  Time 

Productive  Time  for  Each  Component 
In  addition  to  a  detail  sheet  the  user  should  have  a  summary  sheet.    On  this  form  the 
user  can  enter  the  data  from  the  detail  sheets.     From  this  data  the  user  can  then 
determine  the  monthly  run  time. 


1  ? 

Joslin,   Computer  Selection,   p.   83. 
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Alternative  Evaluation  Methods 

The  purpose  of  the  evaluation  process  is  to  define  which  of  the  several  proposals 
submitted  is  best  suited  to  the  organizations  needs.     Determining  which  is  best 
forces  the  user  to  establish  some  criteria   that  can  be  used  as  a  yardstick  to 
measure  the  proposals.     The  objective  of  evaluation   is  usually  to   select  a 
proposed  ADP  system,  capable  of  accomplishing  the  future  workload  at  the  lowest 
total  cost. 

There  are  several  different  ways  for  evaluating  the  proposals.    The  alternatives 
range  from  a  policy  of  not  rocking  the  boat  and  staying  with  the  current  supplier  to  the 
very  competitive  situation  that  uses  objective  criterion  in  the  selection  process.    In 
order  to  serve  the  purposes  of  this  paper  only  the  systems  considered  to  be  objective 
will  be  reviewed. 

The  literature  search  for  this  paper  revealed  three  different  objective  methods 
for  the  evaluation  of  vendor  proposals:  Weighted  Scoring,   Cost-Effectiveness,   and  the 
Cost-Value  techniques.  Cost-Value  was  reviewed  in  Chapter  III. 

Weighted  Scoring 

13 
There  are  several  different  weighted  scoring  systems  that  can  be  used. 


1    ■D 

For  the  interested  reader:    Eugene  S.   Schwartz,   "Computer  Evaluation  and 
Selection,"    Journal  of  Data  Management,   VI,  No.   6(1968),   pp.  58-62;  and,  J.  A. 
Campise,   "A  Quantitative  Approach  to  Equipment  Comparison,  "  Journal  of  Data 
Management,   I,  No.   5(1963),   pp.    12-20. 
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For  this  paper,  the  system  proposed  by  Mr.  Solomon  Rosenthal  will  be  reviewed.  ^ 

The  first  step,  when  using  a  weighted  scoring  system,  is  the  establishment 
of  the  selection  criteria.    The  basis  for  developing  this  plan  is  the  systems 
specification  of  the  information  system.    The  user  must  define  those  characteristics 
of  an  ADP  system  that  will  be  required  to  satisfy  the  needs  of  the  organization.    The 
specific  items  included  and  the  relative  weight  assignment  to  each  item  varies  based 
on  the  requirements  of  the  information  system.     In  Mr.  Rosenthal's  system,  the 
factors  are  classified  as  major,  intermediate,  or  minor. 

The  major  categories  include  factors  such  as:    overall  costs,  equipment 
characteristics,   systems  potential,  vendors  support  and  others  as  required.    The 
intermediate  and  minor  categories  would  include  the  component  elements  that  make 
up  the  major  area.     For  example,  within  the  cost  element,  details  would  focus  on 
the  cost  of  individual  components  and  features  of  each,     hi  addition,   costs  relating 
to  the  installation  of  the  equipment  and  its  subsequent  operation  would  also  be 
considered  in  the  intermediate  or  minor  categories. 

The  next  step  is  to  assign  weights  to  the  factors  to  be  used  to  evaluate  the 
proposals.     It  is  important  to  note  that  if  this  system  is  to  be  impartial,  the  weights 
must  be  assigned  before  any  vendor  proposals  are  received,     hi  fact,  the  weights 
should  be  assigned  before  the  RFP  is  circulated.    The  maximum  score  possible  is 


^Solomon  Rosenthal,  "Analytical  Technique  for  Automatic  Data  Processing- 
Equipment  Acquisition,"    Proceedings  of  the  AFIPS  Spring  Joint  Computer  Conference, 
1964,  pp.   359-366. 
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established  as  100.    This  score  can  be  achieved  only  if  one  proposal  is  the  best  in  all 
factory.    A  part  of  the  100  maximum  is  assigned  to  each  major  factor  based  on  the 
relative  importance  of  each.    Then  weights  are  assigned  to  the  intermediate  and 
minor  factors.     The  sum  of  the  weights  assigned  to  the  intermediate  and  minor 
elements  must  be  equal  to  the  appropriate  major  factor  of  which  they  are  a  part. 
The  results  of  this  process  might  look  as  presented  in  the  following  example. 

FACTORS  WEIGHTS 

Cost    v-  35 

Machine  Rental  20 

One-Time  Costs  10 

Personnel  Support  Cost  5 

Equipment  Capability  25 

Random  Access  Storage  10 

Systems  Validity  Checks  5 

Processing  Speed  10 

System  Potential  15 

Ability  to  Increase  10 

Random  Storage  Rapidly 

Ability  to  Augment 

Central  Processor  5 

Manufacturer  Support  10 

24-Hour  On -Site  Maintenance        5 

Back-up  in  Vicinity  5 

Common  Language  Programming  10 

COBOL  5 

FORTRAN  IV  5 

Physical  Requirements  5 

After  the  proposals  arc  received  and  validated,  they  are  ready  for  scoring.     It 

should  be  noted  that  when  using  this  system  the  individuals  involved  in  the  validation 

of  the  proposals  do  not  do  any  scoring.    The  scoring  is  accomplished  by  a  completely 

different  group  of  people,     hi  addition,  the  scorers  do  not  know  which  vendors' 
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proposal  they  are  processing.      In  fact,  the  scorer  does  not  even  see  the  proposals. 
The  scores  are  recorded  on  an  evaluation  sheet  that  was  prepared  in  the  validation 
process.    The  evaluation  sheet  shows  the  data  applicable  to  each  factor.    In  addition, 
the  validators  prepare  a  written  report  indicating  the  costs,  processing  times,  and 
differences  between  the  proposals  and  the  specifications. 

Based  on  this  information,  the  scoring  begins  with  the  minor  elements  first, 
then  intermediate  scores  are  developed,  followed  by  the  majors  and  finally  the  grand 
total.    The  minor  scores  are  developed  by  either  of  two  methods.     Some  minor  factors 
are  not  compared  one  proposal  witli  another,  but  are  assigned  a  predetermined  score 
if  certain  conditions  exist.     For  example,   if  free  maintenance  is  provided  twenty-four 
hours  per  day,  "X"  points  might  be  awarded,   if  only  for  eight  hours  per  day,  a  lessor 
score  would  be  assigned. 

The  second  method  would  apply  to  the  majority  of  the  minor  elements.     If  the 
smallest  is  best,  the  score  is  computed  as  follows: 


(comparison  base)  (maximum  possible) 

=    minor  score 


(this  vendor  entry) 
If  the  largest  is  best,  the  following  formula  is  used: 


(this  vendor  entry)    (maximum  possible)  .  16 

\ 1?—! __ c 1      -     minor  score  iW 

(comparison  base) 


15Ibid.  16Ibid. 
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The  next  step  is  to  total  the  minor  scores  within  each  intermediate.    The 
intermediate  scores  are  added  together  to  form  the  major  scores  and  the  final  score 
is  the  sum  of  all  the  major  scores. 

The  proposal  then  with  the  highest  score  would  be  recommended  to  top 
management  as  being  die  best  system  to  select. 

Cost  Effectiveness 

The  concept  of  cost  effectiveness  is  primarily  attributable  to  the  work  of  Hitch  and 

McKean ,  * '  and  the  principl  e  s  of  th  i  s  concept  provide  the  foundation  for  a  computer 

selection  system  developed  by  the  Mitre  Corporation.  Prior  to  analyzing  this  system 

however,   it  will  be  helpful  to  define  the  words  cost  and  effectiveness. 

Effectiveness  -  The  degree  to  which  a  system  will  perform  the  future  jobs  and 
satisfy  the  constraints.     Effectiveness  is  generally  considered  to  consist  of  die 
following  three  main  components.   .   .   . 

a.     Capability  -  The  degree  to  which  a  system  will  perform  the  future  jobs  and 
satisfy  the  constraints,  assuming  that  the  system  is  always  available  for  operation 
and  will  never  malfunction.    Capability  can  be  measured  in  various  ways  but  the 
two  key  measures  of  capability  are: 

(1)    Quality  of  the  work  output.    This  measure  is  in  general  multi- 
dimensional since  it  encompasses  the  many  sub  measures  used  to  measure 
the  work  output.     For  example,  it  might  include  the  straighfness  of  a  line  of 
print  or  the  maximum  number  of  copies  of  printout. 


17 

Charles  J.  Hitch  and  Roland  N.   McKean,  The  Economics  of  Defense  in  the 

Nuclear  Age  (Cambridge,   Mass.  :  Harvard  University  Press,    1963). 

1 8J .  D .  Porter  and  B .  II .   Ru dwi ck ,     Application  of  Cost-Effectiveness  Analysis 
to  EDP  System  Selection  (Springfield,  Virginia  Clearinghouse:    U.S.  Department  of 
Coimne7c"e7^AD""667-522,  October,    1967). 
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(2)    Time.    Given  that  the  quality  of  the  work  output  can  be  measured,  a 
second  key  measure  of  the  EDP  system  capability  is  the  time  taken  to 
perform  the  future  work  load,   again  assuming  the  system  is  available  for 
and  never  malfunctions.     For  example,  the  measure  could  be  the  expected 
time  to  perform  a  given  monthly  workload. 

b.  Availability  -  may  be  defined  as  the  probability  that  the  system  will  be  ready 
for  operation  when  called  upon. 

c.  Defindability  -  may  be  defined  as  the  probability  of  the  system  completing 
the  job  satisfactorily,  given  that  it  was  available.   .    .    . 

Cost  -  The  total  dollars  required  to  procure,  operate,  and  maintain  the  system 
to  perform  the  future  set  of  EDP  jobs.   .   .    .  19 

Since  these  two  elements  have  been  defined  it  is  possible  to  measure  each  one 
separately.    The  problem  then  becomes  one  of  combining  these  two  elements  in  a 
way  to  facilitate  decision  making.    To  do  this  it  is  necessary  to  specify  in  advance  the 
level  of  either  the  cost  or  the  level  of  the  effectiveness.    The  mechanics  are  explained 
below  and  illustrated  hi  Figure  7. 

Level  of  effectiveness  specified:    called  pivoting  on  constant  effectiveness.  If 
level  Eo  is  selected  as  the  comparison  level,   system  A  could  not  be  selected.     If  the 
effectiveness  of  A  were  increased  to  Eo  its  cost  would  then  be  at  either  point  A^  or 
An.    Now  system  A  and  B  can  be  compared  with  each  other.    If  the  A  system  cost 
were  A-,  then  it  would  be  selected;  however,   if  the  cost  were  at  A3  then  the  B  system 
would  be  selected. 

Level  of  cost  specified:  called  pivoting  on  constant  cost.     If  the  C2  cost  level  is 
selected  as  the  comparison  level  of  cost,  and  the  cost  of  system  A  is  increased, 
its  level  of  effectiveness  moves  to  either  A2  or  A4.    When  A's  effectiveness  and  cost 


19Ibid.,  p.    13. 
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Effectivenc 


System  B 


Cost 


SOURCE:   J.D.  Porter  and  B.H.   Rudwick,  Application  of  Cost-Effectiveness 
Analysis  to  EDP  System  Selection,  U.S.  Department  of  Commerce,  AD  667-522. 


Figure  7  —  Measurement  of  Cost  and  Effectiveness 
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intersect:  at  A.  the  B  system  is  the  better  buy.  The  converse  is  true  when  the 
effeel  iveness  and  cost  of  the  A  system  is  represented  by  the  point;  A2.  In  this 
situation  the  A  system  would  be  the  better  buy. 

The  first  selection  criterion,  pivoting  on  constant  effectiveness,  is  more 
applicable  to  the  problem  of  computer  selection,   so  the  cost-effectiveness  system 
devised  by  Porter  and  Rudwick  is  based  on  this  principle.     ' 

There  is  a  great  deal  of  similarity  between  the  cost-effectiveness  system  and 
the  cost-value  system  used  by  the  Navy.    Both  methods  are  based  on:  a  probabilistic 
workload  projection,  mandatory  requirements,  and  desirable  features. 

There  are  differences, however  in  the  way  the  projected  workload  is  derived  and 
in  the  way  that  value  is  calculated  for  the  desirable  features.    Both  systems  however 
regard  the  mandatory  items  in  the  same  way.    The  mandatory  features  are  validated 
and  the  requirements  in  the  RFP  concerning  these  items  must  be  satisfied  for  the 
proposal  to  be  considered  responsive. 

The  total  selection  process  using  the  cost-effectiveness  system  is  shown  in 
Appendix  I. 

Probabilistic  workload  projection:    the  basis  for  using  probabilities  in 
projecting  future  workload  is  the  fact  that  the  user  can  predict  with  a  high  degree 

of  confidence   certain   "known  jobs";  however,  there  arc  "likely  jobs"  which  can 

7  1 
only  be  expressed  quantitatively  with  a  lower  degree  of  confidence. 


20Ibid.  ,  p.   17.  21Ibid.  ,  p.   18. 
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The  user  prepares  a  projection  of  his  future  workload  by  starting  out  with  a 
diagram  similar  to  the  one  in  Figure  8.    The  lowest  line  is  called  the  "Reference 
Workload"  and  the  user  has  indicated  that  there  is  a  100  per  cent  probability  that  this 
workload  level  will  be  experienced  or  exceeded.    The  user  then  assigns  a  probability 
to  each  of  the  other  workload  level  projection  lines. 

The  next  step  is  to  average  the  yearly  workload  levels.    This,  then,  permits  a 

diagram  like  that  shown  in  Figure  9  to  be  drawn.    The  purpose  of  this  is  to  simplify 

22 

the  cost  calculations. 

hi  the  validation  phase,  timing  data  was  developed  for  each  system  proposed. 
This  information  is  correlated  with  the  various  workload  levels.     If  benchmarks  were 
used  this  correlation  is  accomplished  by  using  the  extension  factors. 

The  evaluation  team  can  now  construct  a  chart  similar  to  the  one  shown  in 
Figure  10„    In  this  situation  the  RFP  had  a  mandatory  requirement  that  the  workload 
be  completed  in  600  hours.  In  view  of  this,  Vendor  A  has  proposed  two  systems:  A] 
and  A2.     System  A,  ,  a  cheaper  system,  can  satisfy  the  mandatory  requirement  until 
sometime  in  the  third  year  of  the  system  life.    Then  it  is  augmented  to  become  system 
A2,  which  can  satisfy  the  600  hour  requirement  for  the  remainder  of  the  system  life. 

It  will  be  noticed  in  Figure  10  that  the  method  of  expressing  the  probability  is 
different.     The  figures  now  represent  the  probability  of  the  workload  falling  within  the 
respective  segments. 


22 

Porter  and  Rudwick,     Cost  Effectiveness,     p.  23. 
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SOURCE:   J.D.   PorterandB.il.   Rudwick,   Application  of  Cost-Effectiveness 
Analysis  to  EDP  System  Selection,  U.S.  Department  of  Commerce,  AD  667-522. 


Figure  8  —  Probabilistic  Workload  Projection  Smoothed  Over  Time 
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Figure  9  —  Probabilistic  Workload  Projection  Averaged  Po-  Year 
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SOURCE:   J.D.  Porter  and  B.H.  Rudwick,  Application  of  Cost-Effectiveness 
Analysis  to  EDP  System  Selection,  U.S.  Department  of  Commerce,  AD  667-522. 


Figure  10  —  Projected  Time  for  System  A.  and  A.,  Proposed  By  Vendor  A 
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The  cost  of  each  system  can  now  be  calculated.    This  is  done  by  determining 
the  yearly  cost  of  operating  each  system  as  follows: 

C       =       Total  Cost 


P 2     =       Probability  of  workload  in 

segment  1 
C,     =       Cost  of  operation  in  segment  1 


c     =     PiC1+p2c2+.  .  .  .+pncn 

After  the  yearly  cost  is  determined  the  total  discounted  present  value  is  derived  in  a 
manner  similar  to  the  one  shown  in  Appendix  J. 

Desirable  features  make  up  die  third  category  of  the  user's  data  processing 
requirement.    Porter  and  Rudwick  are  concerned  with  the  necessity  of  this  third  way 
of  expressing  data  process  requirements  because: 

1.  The  representative  workload  only  approximates  the  actual  workload. 
Since  there  may  be  other  workload  requirements  that  will  not  be  measured  in  the 
systems  timing  determination,  by  including  desirable  features  the  user  is  provided 
a  hedge  against  this  uncertainty. 

2.  The  inclusion  also  provide  a  hedge  against  the  uncertainty  attendant  to 

O  1 

measuring  a  systems  ability  to  perform  the  future  workload. 


23U.S.  ,  Executive  Office  of  the  President,  Bureau  of  the  Budget,  Discount  Rates 
and  Procedures  to  be  Used  in  Evaluating  Deferred  Costs  and  Benefits,  Circular  A- 94, 
Washington,  D.C.,  Attachment  A. 

24 

Porter  and  Rudwick,  Cost  Effectiveness,  p.    18. 
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3.  The  use  of  desirable  features  permits  a  reduction  in  the  number  of 
mandatory  requirements.    By  definition  the  vendor  must  satisfy  all  mandatory  items 
or  his  proposal  is  eliminated.    This  could  create  an  undesirable  situation.    For 
example,   suppose  the  RFP  contained  a  mandatory  requirement  for  the  proposed 
system  to  have  a  two  second  response  time.    Vendor  A's  system  has  a  response 
time  of  2.  1  seconds,  but  that  system  costs  20  per  cent  less  than  all  others.     It  is 
difficult  to  defend  the  need  for  that  additional  one-tenth  of  a  second  advantage, 
particularly  in  view  of  system  A's  cost. 

4.  The  manufacturers  of  data  processing  equipment  are  competing  with  each 
other.    This  spawns  technological  innovations  which  improve  the  design  of  computers 
and  the  attendant  software.    The  use  of  desirable  features  in  the  RFP  provides  the 
vendor  an  excellent  avenue  for  proposing  these  advances  to  the  user.    Without  these 
features  the  user  would  be  denied  the  opportunity  of  evaluating  these  advanced 
capabilities. 

Once  the  desirable  features  have  been  stated  it  is  then  necessary  to  provide  a 
way  to  evaluate  the  vendor  proposals.    There  has  to  be  a  way  to  relate  the  character- 
istics of  any  particular  desirable  feature  to  the  job  or  jobs  that  will  be  improved  by 
the  use  of  this  desirable.    The  question  then  simply  stated  is:    Is  the  desirable  feature 
offered  by  the  vendor  worth  the  price  he  charges?     This  question  cannot  be  answered 
unless  a  value  of  this  worth  has  been  assigned.     In  order  to  assign  a  value,  however, 
the  term  worth  has  to  be  defined.     In  this  system,  the  worth  of  a  feature  is  equal  to 
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"the  lowest  incremental  cost  to  do  the  same  job  if  the  feature  is  not  available.  "^^ 

There  are  two  ways  provided  for  determining  and  evaluating  the  worth  of  any 
desirable  feature.    The  first  way  determines  worth  through  an  analytical  process 
and  the  second  method  makes  use  of  comparative  ranking. 

Analytical  determination  of  worth:    Since  the  worth  of  a  feature  is  defined  as 
the  lowest  incremental  cost  of  doing  the  job  if  the  desired  feature  is  not  available 
from  the  suppliers,  it  is  necessary  to  first  identify  the  alternative  ways  of  doing  the 
job.     Figure  11  depicts  the  alternatives  that  are  available  to  the  user.    The  next 
step  is  to  determine  the  cost  over  the  life  of  the  system  of  alternative  ways  of 
accomplishing  the  function.    The  worth  of  the'fea  t  ure  is  the  least  of  these  costs. 

It  is  also  necessary  to  identify  the  cost  of  the  alternative  ways  of  obtaining  the 
feature  and  doing  the  job  using  the  feature.     For  example  the  feature  might  be  procur 
procured  from  other  sources  or,  in  the  case  of  software  the  user  might  develop  the 
capability  using  in-house  resources. 

Based  on  this  compilation  of  data,  it  is  then  possible  to  determine  the  way  to 
accomplish  the  function  at  the  lowest  total  cost.    This  is  referred  to  as  the  "efficient" 
solution.        Restated,  the  efficient  solution  is  found  by  determining  the  lowest  cost 
method  of  obtaining  the  feature  and  doing  the  job,  then  comparing  this  cost  with  the 
worth,  and  choosing  the  lowest  cost  alternative. 


25lbid.  ,  p.   33.  26I!)id.  ,  p.   34 
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SOURCE:   J.D.  Porter  and  B.H.   Rudwick,  Application  of  Cost-Effectiveness 
Analysis  to  EDP  System  Selection,  U.S.  Department  of  Commerce,  AD  667-522. 


Figure  11  —  Alternatives  Available  to  the  User 
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Determination  of  worth  by  comparative  ranking:    There  may  be  times  when  it 
will  not  be  possible  to  determine  the  worth  of  all  desirable  features  by  analysis  and 
considered  judgement,    hi  these  instances,  intuitive  judgement  is  substituted  as  a 
part  of  the  quantitative  evaluation.    The  steps  to  implement  this  procedure  are: 

(a)  All  desirable  features  are  ranked  in  order  of  importance. 

(b)  Points  are  allocated  to  each  feature  thus  establishing  the  relative  worth  of 
each. 

(c)  The  points  are  translated  into  dollars.    This  is  done  by  calibrating  one  or 
more  of  the  features  by  determining  its  worth  on  the  analytical  basis  described 
above . 

(d)  The  results  derived  are  reviewed  using  intuitive  judgement  as  the  sounding 
board.     If  discrepancies  are  found  then  either  all  or  some  of  the  items  should  go 
through  the  whole  process  again.  27 

It  is  now  time  to  bring  together  the  information  derived  in  the  various  areas, 
so  that  a  composite  picture  of  each  proposal  -can  be  developed.    To  do  this  a 
worksheet  like  the  one  shown  in  Figure  12  will  be  used.     In  this  example  there 
are  three  vendors'  (A,  B,  and  C)  proposals  being  evaluated.    There  are  three 
desirable  features  to  be  considered— Fi,   F2,  and  F3.    The  users  worth  and  the 
least  cost  for  each  is  shown  in  item  3  on  the  Evaluator  Worksheet  (Figure  12). 
This  illustration  demonstrates  how  the  costs  are  derived  if  the  vendor  does  not 
provide  the  feature  at  all,  or  the  feature  is  provided  in  the  basic  system  and  cannot. 
be  excluded,  and  finally  the  instance  where  the  vendor  does  propose  the  feature  and 
states  its  cost. 

The  initial  entry  on  the  worksheet  is  the  Total  Proposed  Vendor  Cost.    This 

27Ibid.  ,  p.  37. 
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Cost  Elements 

System  Cost 

1.      Total  Proposed  Vendor  Cost 

CA 

cB 

CC 

300K 

3  J  OK 

330K 

2.      Expected  Cost  to  Do  Representative  Workload 

300K 

305  K 

31  OK 

3.      Cost  of  Additional  Job  Benefits 

10K 

~ 

10K 

Desirable  User 

Least 
Cost 

Vendor  Cost 

Feature 

Worth 

ca 

"cB 

cc 

Fl 

10K 

15K 

- 

incl 

15  K 

172 

25K 

20K 

- 

_ 

5K 

2  OK 

2  OK            5K 

F3 

10K 

20K 

« 

5K 

incl 

10K 

5K 

4.      Total  Expected  Cost  To  Do  User  Job 

34  OK 

330K        325Ka 

i 

Vendor  C  selected  -  lowest  total  cost. 


SOURCE:   J.D.  Porter  and  B.  H .   Rudwick,   Application  of  Cost -Effectiveness 
Analysis  to  EDP  System  Selection,  U.S.  Department  of  Commerce,  AD667-522. 


Figure  12  —  Evaluator's  Worksheet 
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represents  the  cost  of  each  manufacturers  proposed  system,  if  the  system  is 
accepted  as  proposed.     It  also  is  the  cost  as  calculated  by  the  vendor. 

The  evaluator  enters  the  previously  derived  cost  data  to  process  the  workload 
of  the  life  of  the  system,  and  thus  derives  the  expected  cost  to  do  a  representative 
workload. 

At  this  time  the  desirable  features  which  each  vendor  has  proposed  and  the 
incremental  costs  associated  with  each  are  entered  on  the  worksheet  to  determine 
the  cost  of  additional  job  benefits.     In  this  example  note  that  vendor  A  does  not 
provide  any  of  the  three  features,    hi  the  case  of  vendor  B,  feature  F^  is  included  in 
the  system  cost  while  F2  is  not  provided  but  F3  is  at  the  cost  indicated.    Vendor  C 
has  provided  all  of  the  features.    The  cost  of  feature  Fo  however,   is  not  quoted 
separately  but  included  in  the  system  cost.    Now  the  evaluator  can  determine  for 
each  vendor  the  least  costly  of  the  three  alternative  ways  of  receiving  the  benefits 
provided  by  each  of  the  desirable  features.    The  three  alternatives  are: 

a.  Buying  the  desirable  feature  from  the  vendor  (at  the  vendor's  proposed  cost). 

b.  Obtaining  the  desirable  feature  from  another  source  (at  the  least  cost  of 
feature  if  obtained  separately). 

c.  Not  buying  the  feature,  but  using  the  least  costly  alternative  way  to  provide 
the  benefits  (at  a  cost  equal  to  user  worth).  28 

The  evaluator  now  enters  the  values  representing  Users  Worth  and  the  Least 

Cost.    These  values  are  calculated  prior  to  the  actual  evaluation  process. 

The  lowest  additional  user  cost  for  obtaining  the  desirable  feature  can  be 


2°Porter  and  Rudwick,  Cost  Effectiveness,  p.  45. 
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determined.    Note  in  Figure  12  the  cost  data  entered  for  the  three  desirable  features, 

In  the  case  of  F]  the  User  Worth  is  $10K,  that  is  the  user  can  perform  the  jobs 
associated  with  this  feature  at  an  expected  cost  of  $10K.     Since  vendor  A  does  not 
provide  this  feature,  the  user  will  be  forced  to  spend  $3 OK  in  addition  to  vendor  A 
costs  to  satisfy  the  requirements  associated  with  Fj.    Vendor  B  includes  this  feature 
as  part  of  his  basic  system  and  the  cost  of  it  cannot;  be  segregated.    The  user  then 
will  not  have  to  spend  the  $10K  if  vendor  B's  system  is  used.    Vendor  C  can  provide 
Fj_  however,  the  cost  is  $15K.    Thus,  the  evaluator  would  eliminate  the  feature  from 
vendor  C's  proposal  since  its  cost  is  higher  than  its  worth  to  the  user.    This  same 
type  of  analysis  is  applied  to  all  of  the  other  desirable  features. 

The  total  expected  cost  to  the  user  is  then  calculated  by  adding  the  cost  of 
the  desirable  feature,  or  the  user  cost  equivalent,  to  the  expected  cost  to  do  the 
representative  workload.    This  cost  completes  the  cost  calculation. 

There  are  some  points  worth  noting  concerning  this  method  of  arriving  at  the 
final  cost  figures  for  the  proposals.    Vendor  A  had  the  lowest  proposed  cost  and  die 
lowest  cost  to  perform  the  representative  workload.    On  the  other  hand,  winning- 
vendor  C  had  the  highest  proposed  cost  and  the  highest  cost  to  perform  the  workload. 
These  costs  however,  do  not  cover  all  of  the  users  requirements.    The  total  cost  is 
the  only  one  that  will  provide  for  satisfying  the  total  data  processing  requirement  of 

the  user.    Accordingly  the  total  cost  figures  are  the  true  criterion  for  making  the 

i      •    •        29 
decision. 


29 

Ibid.  ,  p.  46. 


CHAPTER  V 
EVALUATION  OF  THE  NAVY  SYSTEM 

It  is  the  purpose  of  this  chapter  to  evaluate  the  Navy's  competive  computer 
selection  system.    The  review  will  be  conducted  by  first  appraising  the  organization 
responsible  for  selection  and  then  by  looking  at  the  various  stages  in  the  process. 

The  Automatic  Data  Proces s i ng  Equ ipment  Selection  Office 

The  Secretary  of  the  Navy  established  ADPESO  on  1  July  1967.     It  is  essential 
to  appreciate  the  role  played  by  this  organization  if  one  is  going  to  evaluate  the 
Navy's  computer  selection  system.    This  becomes  abundantly  clear  when  one 
considers  the  fact  that  prior  to  the  establishment  of  ADPESO  there  was  no  one 
system  in  use.    The  selection  of  ADPE  prior  to  ADPESO  was  accomplished  by 
the  various  heads  of  departmental  components.    The  procedures  used  varied 
depending  on  the  philosophy  of  each  of  the  departmental  components.    There  was 
widespread  use  of  ad  hoc  committees  that  were  formed  as  the  need  arose.    The 
contracting  was  conducted  by  a  regular  contracting  officer,  not  one  specialized  in 
the  procurement  of  ADPE. 
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The  establishment  of  ADPESO  thus  accomplished  two  very  significant  things. 
First  the  responsibility  for  selection  of  ADPE  was  centralized  in  the  Navy.     Secondly 
a  full  time  staff  was  recruited  to  carry  out  the  mission  of  ADPESO. 

In  addition  to  centralizing  the  function,  the  function  was  also  elevated  to  a 
higher  level  in  the  Department  of  the  Navy.    The  Director  of  ADPESO  reports  to  the 
Special  Assistant  to  the  Secretary  of  the  Navy.    Thus,  top  management  attention  is 
always  focused  on  this  vital  function. 

There  are  twenty- two  professional  members  on  the  staff  of  ADPESO.    These 
individuals  arc  responsible  for  developing  the  procedures  used,  plus  performing  the 
functions  of  equipment  selection.    There  are  several  significant  benefits  derived  from 
this.     First,  and  probably  most  significant,  is  the  fact  that  these  individuals  are 
professionals.    This  means  that  the  breadth  and  depth  of  knowledge  of  each  individual 
presently  involved  in  the  selection  process  is  many  times  greater  now  than  was  the 
case  prior  to  July  1967.    This  increased  knowledge  has  a  very  profound  impact  on  the 
RFP's  sent  to  vendors  and  the  appraisal  of  the  proposals  submitted  by  the  suppliers. 

In  the  years  prior  to  the  establishment  of  ADPESO,  the  specifications  and  RFP's 
were  prepared  by  the  user  activities.    The  degree  to  which  these  specifications  were 
not  oriented  toward  any  given  manufacturer  was  dependent  on  the  experience  of  the 
individuals  preparing  them.     Since  these  individuals  were  normally  not  involved  fuli 
time  in  the  selection  process,  their  range  of  knowledge  was  limited.     In  view  of  this 
it  was  possible  for  the  specifications  to  be  oriented  one  way  or  another.    The  full  time 
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professional  staff  at  ADPESO  does  not  labor  under  this  handicap  so  the  package  sent 
to  the  vendors  is  not  susceptible  to  this  flaw. 

When  the  vendor's  proposals  are  received,  the  full-lime  professional  staff  is 
responsible  for  reviewing  them.     From  time  to  time,  user  personnel  do  assist  in  the 
appraisal  process,  but  this  effort  is  still  under  the  close  control  and  direction  of 
ADPESO.    As  indicated  in  Chapter  III,  the  group  actually  responsible  for  preparation 
of  the  RFP  and  the  review  of  all  the  corresponding  proposals  is  the  Source  Selection 
Evaluation  Board  (SSEB\.-     It  is  important  to  note  that  the  SSEB  is  constituted  for 
the  entire  life  of  equipment  acquisition.    This  means  the  individuals  can  become 
intimately  acquainted  with  the  requirements  of  the  individual  cases.     This 
comprehensive  understanding  of  the  user  requirement  coupled  with  specialized 
knowledge  of  the  individuals  on  the  SSEB  create  a  climate  favorable  to  rational  and 
objective  evaluations. 


Case 

GSA 
Price 

ADPESO 

Negotiated 

Price 

Dollar 

Saviugs 

% 
Discount 

A 

$3,300,000 

$2,600,000 

$     700,000 

21% 

B 

4,743,612 

4,327,981 

415,631 

9% 

C 

5,861,976 

3,752,796 

2,190,180 

37% 

SOURCE 

:    Information  provided  by  Mr.  A.E.  Feenan,  Head, 

Sp< 

deification  and  Proposal  Division,  Automatic  Data 

Pr< 

^cessing  Equipment  Selection  Office. 

Figure  12  —  Navy  Savings  by  ADPESO 
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The  contracting  officer  is  a  key  ADPESO  employee.    He  devotes  all  his  efforts 
to  contracting  for  ADPE.     By  specializing  in  the  procurement  of  ADPE,  this  individual 
has  developed  an  in-depth  knowledge,  understanding,  and  appreciation  for  the 
particular  problems  associated  with  acquisition  of  the  equipment.    The  results  are 
substantial  savings  for  the  Navy  as  shown  in  the  cases  presented  in  Figure  12. 

Evaluation  of  Selection  Procedures 

In  Chapter  III  the  Navy's  selection  process  was  divided  into  five  subcategories. 
Accordingly,  the  evaluation  of  the  system  will  be  done  by  examining  each  of  these 
areas. 

Systems  Specifications 

First,  it  is  significant  to  point  out  that  prior  to  the  establishment  of  ADPESO, 
there  was  no  standard  method  by  which  a  user  could  document  his  data  processing 
requirement.     Each  level  in  the  user's  chain- of- command  had  its  own  unique  set  of 
requirements  concerning  what  information  should  be  included  and  the  format  in  which 
the  specifications  were  to  be  presented.    Accordingly  ADPESO' s  most  important 
initial  contribution  to  computer  selection  in  the  Navy  was  to  promulgate  instructions 
that  standardized  the  format  of  the  specifications.  1 


*E.  O.  Joslin,  "Competitive  Computer  Selection  Within  the  Department  of  the 
Navy,"  (Unpublished  paper  presented  to  the  Dicbold  Group,   Inc.  ,  October,    1969), 
p.  5. 
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The  specification  review  conducted  by  ADPESO  is  very  extensive.  This  review 
plus  the  standardized  format  provides  the  Navy  with  an  objective  set  of  requirements 
for  submission  to  the  equipment  suppliers. 

It  is  significant  to  note  that  it  is  the  Navy's  policy  to  minimize  the  number  of 

9 

mandatory  requirements  contained  in  the  specifications.       Every  item  that  is 
classified  as  mandatory  is  carefully  reviewed.     When  the  justification  does  not 
substantiate  the  claim  the  item  is  reclassified  as  a  desirable  feature.     This  is 
done   so  that  the   specifications  will  not  become   so  stringent  as  to  either  discourage 
vendor  participation  or  to  cause  an  undue  number  of  proposals  to  be  classified  as 
non-  re  spon  sive . 

The  Selection  Plan 

The  selection  plan  consists  of  the  detailed  instructions  that  will  be  used  to 
evaluate  the  proposals  submitted  by  the  vendors.    The  significant  feature  of  the 
Navy's  philosophy  here  is  that  die  plan  must  be  prepared  before  the  RFP  is  issued.  3 
This  means  that  greater  objectivity  is  injected  into  the  evaluation  process.     If  the 
plan  were  not  prepared  until  after  the  initial  review  of  the  proposals,  there  would 
be  danger  of  personal  preference  creeping  into  the  plan. 


2(J.  S.  ,  Department  of  the  Navy,  "Specifications,  Selection  and  Acquisition  of 
Automatic  Data  Processing  Equipment,   Secretary  of  the  Navy  List  ruction  10462.  13, 
Department  of  the  Navy,  Washington,  D.C.  20390,  p.  B-ll. 

3  Ibid.  ,  p.   Ill- 5. 
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Vendor  Liaison 

The  ADPESO  has  defined,  some  very  meaningful  rules  that  govern  the 
relationship  between  its  representatives  and  the  various  manufacturers.    The  intent 
of  these  rules  is  to  establish  an  impartial  and  objective  way  to  deal  with  the  vendors. 
These  rules  have  been  provided  to  the  vendors  so  that  they  know  how  to  proceed  in 
their  dealings  with  the  Navy  personnel.    One  very  noteworthy  policy  is  that 
information  provided  to  one  supplier  is  provided  to  all  and  in  writing.    The 
comprehensiveness  of  the  policies  concerning  supplier  relations  plus  the  fact 
that  ADPESO  is  a  continuing  body  staffed  with  professionals  creates  a  climate  for 
fair  and  objective  interface  between  the  vendors  and  the  Navy. 

Validation  of  Vendor  Proposals 

The  validation  of  the  vendor's  proposals  is  concerned  with  two  areas.     First  is 
the  verification  of  the  proposed,  systems  capabilities  and  die  second  is  validation  of 
the  system  timing.    The  Navy  uses  several  sources  of  data  to  verify  the  systems 
capabilities.     First,  the  vendors  are  requested  to  provide  technical  data  in  their 
proposal.     In  addition  ADPESO  has  manufacturer  tedinical  manuals  and  die  Auerbach 
Standard  EDP  Reports.    There  is  however,  a  factor  that  seems  to  be  even  more 
important  than  these  aids- -the  professional  staff  at  ADPESO.    Because  these  people 
are  continually  dealing  with  the  subject  matter,  their  ability  to  appraise  the  vendor 
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claims  is  certainly  greater  than  what  could  have  been  expected  prior  to  the 
establishment  of  ADPESO. 

The  primary  method  of  validating  the  system  timing  is  through  benchmark 
programs.     In  Chapter  IV  two  other  methods  of  system  timing  were  discussed- 
computer  simulation  and  hand  timing.    There  are  advantages  and  disadvantages  to 
each  of  these  methods.     In  certain  instances  it  is  possible  that  any  one  would  be  the 
best  considering  all  factors.    Joslin  believes  however,  that  the  prime  consideration  in 
choosing  a  timing  technique  should  be  the  degree  of  confidence  required  in  the  data 
derived.    The  following  chart  indicates  the  levels  of  confidence  that  can  be  obtained 
with  the  three  methods.    Notice  that  the  confidence  levels  are  shown  covering  a  wide 
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range.    This  is  necessary  to  allow  for  variations  in  the  applications  and  personnel 
using  the  techniques.    The  information  indicates  that  the  highest  levels  of  confidence 
can  be  placed  in  the  results  obtained  from  the  use  of  a  benchmark  program. 

There  is  another  significant  advantage  of  using  benchmark  programs — the 
programs  are  run  on  the  system  proposed  by  the  vendor.    This  gives  the  user  an 
opportunity  to  determine  the  effect  on  the  application  of  such  features  as:    I/o  control, 
central  processor  administrative  and  processing  functions,  effective  speed  of  the 
various  I/o  devices,   system  simultaneity,  the  software  scheduler,  and  the  efficiency 
of  the  programming  system  in  using  the  hardware  capabilities. 

Another  very  important  item  to  consider  is  the  cost  element.    The  U.S.  Army 
in  1968,  conducted  a  study  designed  to  determine  the  advisability  of  continuing  an 
effort  to  program  a  computer  simulator.     In  the  course  of  this  review,   cost  data  was 
developed  concerning  three  timing  techniques.    The  Army  found  the:  cost  per  project 
for  obtaining'  timing  data  was: 

Hand  Timing  $34,124, 

Simulation  $78,532,  and 

Benchmarks  $29, 548.  6 

4  , 
E.O.  Joslin,  Computer  Selection   (Reading,  Massachusetts:  Addison -Wesley 

Publishing  Company,    1968^),  p.   105. 

5 lb id.  ,  p.  78. 

U.  S.  ,  Department  of  the  Army,   Computer  System  Support  and  Evaluation 
Command,   Report  of  the  S^  Study  Group  System  and  Software  Simulator  (Washington, 
D.  C. :  December  20,   1968). 


87 


Based  on  this  plus  other  considerations  in  the  study,  the  Army  decided  not  to 
continue  developing  a  simulator.     Instead,  benchmark  programs  are  being  used. 

There  are  disadvantages  to  using  the  benchmark  programs  however. 
Before  benchmark  programs  can  be  used  the  actual  programs  have  to  be  written. 
This  task  takes  time.    The  amount  of  time  varies  with  the  complexity  of  the 
programs,  the  knowledge  and  experience  of  the  programmers,  particularly  the  level 
of  experience  with  programming  languages  such  as  COBOL,   FORTRAN,  or  JOVIAL. 
This  is  so  because  these  languages  generally  can  be  processed  by  all  the  different 
computers.    This  time  element   could  cause  the  user  a  loss  of  a  state-of-art  condition. 
The  influence  of  this  factor  should  decrease  within  the  Navy  in  the  future  for  the 
trend  is  toward  using  these  higher  level  languages  as  a  matter  of  course.      In  light 
of  the  advantages  stated  perviously,  plus  the  fact  that  the  DOD  now  requires  all 
services  to  use  benchmarks,  it  appears  that  the  Navy's  use  of  this  technique  makes 
the  best  of  a  difficult  task. 

Evaluation  of  Vendor  Proposals 

The  method  used  by  the  Navy  to  evaluate  vendor  proposals  is  based  on  the  Cost- 
Value  Technique.       In  Chapter  IV,  two  other  evaluation  systems  were  reviewed — 
Cost-Effectiveness  and  the  Weighted  Scoring  methods.    There  is  a  great  deal  of 


7 

Joslin,   Computer  Selection,  p.    18. 
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similarity  between  the  Cost-Value  and  Cost-Effectiveness  systems.     One  difference 
is  the  way  the  worth  of  a  desirable  feature  is  derived.     Another  difference,  only  slight 
however,   is  in  the  way  the  workload  projection  is  handled. 

The  Cost-Effectiveness  system  does  not  provide  for  the  probability  of  the 
workload  falling  below  the  projected  reference  workload  line.     In  the  Navy's  system, 
the  user  indicates  a  projected  rate  of  growth  and  then  assigns  a  probability  to  the 
levels  above  and  below  the  trend  line.    This  approach  seems  to  be  more  realistic. 

The  significant  advantage  of  the  Cost-Value  system  versus  the  Weighted 
Scoring  method,  particularly  to  top  management,  is  that  the  results  are  expressed  in 
dollars  instead  of  points.    By  expressing  the  evaluation  of  each  vendor  proposal  hi 
dollars,  the  senior  managers  can  readily  understand  what  is  happening  in  the 
evaluation  process,  but;  there  is  no  real  need  for  these  people  to  understand  complex 
and  technical  data  concerning  each  proposal. 

Another  important  characteristic  of  the  Cost- Value  technique  not  found  in  the 
Weighted  Scoring  system  is  the  ability  of  the  user  to  analyze  cost  associated  with 
desirable  features.    The  user  can  look  at  worth  and  cost  of  a  desirable  feature  and 
make  a  sound  procurement  decision.    This  capability  is  particularly  important  in  light 
of  the  new  pricing  policies  of  ADPE  manufacturers. 

While  the  Weighted  Scoring  system  appears  to  be  objective,  there  is  a  very 
significant  crack  in  its  wall  of  objectivity.    The  factors  chosen  as  the  evaluation 
criteria  and  the  assignment  of  weights  to  them  is  subjective  not  objective.    There  are 
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no  objective  guidelines  to  assist  buyers  in  matching  requirements  with  factors  or  the 
relative  importance  of  each.     It  is  unlikely  that  any  two  analysts,  given  the  job  of 
independently  establishing  appropriate  factors  and  weights  to  select  the  best  computer 
for  a  particular  installation,  would  arrive  at  similar  conclusions.    There  also  is  a 
possibility  that  the  weights  could  be  juggled  and  thus  lead  to  almost  any  result. 
Further,  there  is  a  real  danger  that  by  the  time  the  analyst  has  performed  all  the 
necessary  weighting  and  scoring,  he  might  lose  sight  of  their  shaky  subjective 
foundation  and  attach  undue  significance  to  the  results. 

In  view  of  the  above  factors,   it  is  believed  that  the  Cost -Value  technique  is  the 
best  method  of  evaluating  the  vendor's  proposals. 

Suggested  Improvement 

While  the  Navy's  selection  system  is  excellent,  there  is  one  area  that  could  be 
made  even  more  meaningful.     Currently  the  method  of  costing  the  vendor  proposal 
results  in  a  figure  that  is  the  most  probable.    This  figure  is  derived  by  the  method 
described  in  Chapter  III.    Basically  the  calculations  involve  a  monthly  workload  cost 
at  various  levels  plus  the  probability  of  operating  at  these  levels  and  a  discount  factor. 

Mr.  David  Hertz  proposes  a  way  of  informing  the  executive  of  the  uncertainity 
surrounding  capital  investment  decisions.       In  this  system,   probabilities  are  assigned 


8John  R.  Hillcgass,  "Systematic  Technique  For  Computer  Evaluation  and 
Selection,"  Management  Services,  VI  (Jul y- August,   1969),  p.   39. 

9DavidB.  Hertz,  "Risk  Analysis  in  Capital  Investment,"  Harvard  Business 
Review,  XXXXI1  No.    1  (January/February  196-1),  p.  95. 
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the  various  possible  outcomes.    The  cash  flows  connected  with  each  of  these 
possibilities  is  calculated.    These  values  are  then  used  to  determine  the  expected 
net  present  value.    This  data  is  then  portrayed  graphically.    The  significant 
advantage  of  this  is  that  it  allows  management  to  see  the  expected  cash  flows  plus 
the  variability  of  the  flows  and  the  risk  associated  with  each.    The  present  system 
contains  all  of  the  data  required  to  perform  such  an  analysis.     It  would  be  necessary 
however,  to  obtain  a  computer  program  to  perform  the  required  calculations.     If  this 
type  of  analysis  were  incorporated  into  the  Navy  system  it  is  believed  that  there  would 
be  substantial  benefits  derived.     Accordingly  it  is  recommended  the  Navy  conduct  a 
study  and  investigate  the  feasibility  of  using  a-risk  analysis  technique  in  its  evaluation 
process. 

Since  ADPESO  was  established,  experience  with  centralized  competitive 
selection  of  computer  systems  in  the  Navy  has  demonstrated  that:    the  using  activities 
are  happier,  the  vendors  are  happier  because  they  feel  they  are  being  given  a  true 
chance  to  bid,  and  the  cost  of  selection  within  the  Department  of  the  Navy  has  been 
reduced  50  per  cent  while  the  number  installations  has  increased  30  per  cent.    Thus, 
centralized,   competitive  computer  selection  in  the  Navy  has  proven  to  be  very 
satisfactory.  10 


Joslin,  "Competitive  Computer  Selection  Within  the  Department  of  die  Navy," 


p.   35. 


CHAPTER  VI 


CONCLUSION 


The  purpose  of  this  paper  has  been  to  appraise  the  competitive  computer 
selection  system  used  by  the  U.S.  Navy.     First,   the  Navy's  system,  was  defined  and 
then  alternative  methodologies  were  reviewed  in  order  to  have  a  basis  for  the 
appraisal. 

The  Navy's  competitive  computer  selection  system  is  a  highly  regulated  process. 
There  are  many  regulations  and  policies  issued  by  higher  authorities  such  as  the 
Congress,  the  President,  the  Secretary  of  Defense  and  others. 

The  Navy  docs  however,  have  a  great  deal  of  latitude  in  deciding  exactly  which 
methodologies  will  be  used  to  differentiate  among  vendor  proposals. 

The  Automatic  Data  Processing  Equipment  Selection  Office  is  the  organization 
within  the  Department  of  the  Navy  charged  with  the  responsibility  for  the  development 
and  administration  of  the  Department  of  the  Navy  plans,  policies,  procedures,  and 
methods  governing  ADPE  specification  preparation,  proposal  solicitation,  proposal 
evaluation  and  selection. 

The  ADPESO  is  located  in  the  office  of  the  Special  Assistant  to  the  Secretary  of 
the  Navy,  who  is  designated  as  the  senior  ADP  policy  official  in  the  Navy. 
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The  Navy's  competitive  computer  selection  process  consists  of  five  major 

activities: 

Specification  Review 

Preparation  of  the  RFP  and  Selection  Plan 

Vendor  Liaison  and  Validation  of  Proposals 

Proposal  Evaluation 

Contract  Negotiation  and  Award 

The  timing  of  the  proposed  computer  configurations  is  validated  by  using 
benchmark  programs.     In  conjunction  with  these  programs,  a  probabilistic  workload 
projection  technique  is  used  to  define  the  monthly  cost  to  perform  the  workload.    The 
monthly  costs  are  consolidated  into  yearly  increments  to  arrive  at  the  annual  costs 
which  in  turn  are  discounted  to  represent  the  present  value  of  the  total  cost  to 
perform  the  workload  over  the  total  system  life. 

The  Cost- Value  Technique  is  used  to  evaluate  the  desirable  features  offered  by 
the  vendors.    The  distinguishing  feature  of  this  technique  is  die  assignment  of  value 
to  desirable  features  in  terms  of  dollars.    Thus,  an  important  benefit  derived  from 
the  use  of  this  method  is  improved  management  understanding  of  the  evaluation 
process,  thereby  facilitating  meaningful  decisions  relative  to  the  value  of  any  or  all 
of  the  desirable  features. 

By  contrast,  die  Navy  could  validate  the  timing  of  the  proposed  systems  by 
calculating  the  times  manually  or  by  using  a  computer  simulation  routine  such  as 
SCERT.    These  methods  however  do  not  provide  data  that  is  as  reliable  as  the  bench- 
mark programs.     In  addition,   a  cost  study  conducted  by  the  U.S.  Army  found  that 
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benchmark  programs  could  be  prepared  and  used  at  a  lower  cost  than  the  other  two 
techniques. 

The  Cost- Effectiveness  and  Weighted  Scoring  systems  are  alternative  processes 
that  can  be  used  to  evaluate  vendor  proposals.    The  Cost- Effectiveness  and  Cost- 
Value  systems  are  very  similar.    There  is  a  slight  difference  in  assigning  worth  to 
desirable  features.    There  is  a  marked  difference  between  the  Cost- Value  and  the 
Weighted  Scoring  techniques.    The  most  significant  is  that  Cost- Value  uses  dollars 
as  an  evaluation  scheme,  while  the  latter  awards  points  based  on  a  predefined  point 
scoring  templet.    These  scores  are  added  up,  and  the  system  with  the  most  points 
ostensibly  is  the  best  choice. 

The  flaw  in  Weighted  Scoring  system  is  that  the  factors  to  be  evaluated  and  the 
weights  assigned  to  them  are  by  necessity,  chosen  arbitrarily.    There  are  no 
objective  guidelines  for  matching  factors  and  weights  to  a  particular  users  needs. 

The  Author  has  made  one  recommendation  to  improve  the  Navy  system.     It  has 
been  suggested  that  a  methodology  advocated  by  David  Hertz,   called  Risk  Analysis 
be  incorporated  in  the  system. 
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To  provide  for  the  economic  and  efficient  purchase,  lease,  maintenance,  opera- 
tion, and  utilization  of  automatic  data  processing  equipment  by  Federal 
departments  and  agencies. 

Be  it  enacted  by  the  Senate  and  limine  of  Representatives  of  the 
United  States  of  Am-erica  in  Congress  assembled.  That  title  1  of  the 
Federal  Property  and  Administrative  Service  Act  of  1949  (G3  Stat. 
377),  as  amended,  is  hereby  amended  by  adding  a  new  section  to  i  1 
as  follov,  s: 

"automatic  DATA  PROCESSING  equipment 

"Sec.  111.  (a)  The  Administrator  is  authoi-ized  and  directed  to 
coordinate  and  provide  for  the  economic  and  efficient  purchase,  lease, 
and  maintenance  of  automatic  data  processing  equipment  by  Federal 
agencies. 

"(b)  (1 )  Automatic  data  processing  equipment  suitable  for  efficient 
and  effective  use  by  Federal  agencies  shall  be  provided  by  the  Adminis- 
trator through  purchase,  lease,  transfer  or  equipment  from  other 
Federal  agencies,  or  otherwise,  and  the  Administrator  is  authorized 
and  directed  to  provide  by  contract  or  otherwise  for  the  maintenance 
and  repair  of  such  equipment.  In  carrying  out  iTis  responsibilities 
under  this  section  the  Administrator  is  authorized  to  transfer  auto- 
matic data  processing  equipment 'between  Federal  agencies,  to  provide 
for  joint  utilization  of  such  equipment  by  two  or  more  Federal 
agencies,  and  to  establish  and  operate  equipment  pools  and  data 
processing  centers  for  the  use  of  two  or  more  such  agencies  when 
necessary  for  its  most  efficient  and  effective  utilization. 

"(2)  The  Administrator  may  delegate  to  one  or  more  Federal 
agencies  authority  to  operate  automatic  data  processing  equipment 
pools  and  automatic  data  processing  centers,  and  to  lease,  purchase, 
or  maintain  individual  automatic  data  processing  systems  or  specific 
units  of  equipment,  including  such  equipment  used  m  automatic  data 
processing  pools  and  automatic  data  processing  centers,  when  such 
action  is  determined  by  the  Administrator  to  be  necessary  for  the 
economy  and  efficiency  of  operations,  or  when  such  action  is  essential 
to  national  defense  or  national  security.  The  Administrator  may 
delegate  to  one  or  more  Federal  agencies  authority  to  lease,  purchase, 
or  maintain  automatic  data  processing  equipment  to  the  extent  to 
which  he  determines  such  action  to  be  necessary  and  desirable  to  allow 
for  the  orderly  implementation  of  a  program  for  the  utilization  of 
such  equipment. 

"(c)  There  is  hereby  authorized  to  be  established  on  the  books 
of  the  Treasury  an  automatic  data  processing  fund,  which  shall  be 
available  without  fiscal  year  limitation  for  expenses,  including  per- 
sonal services,  other  costs,  and  the.  procurement  by  lease,  purch 
transfer,  or  otherwise  of  equipment,  maintenance,  and  repair  of  such 
equipment  by  contract  or  otherwise,  necessary  for  the  efficient  coordi- 
nation, operation,  utilization  of  such  equipment  by  and  for  Federal 
agencies:  Provided,  That  a  report  of  equipment  inventory,  utiliza- 
tion, and  acquisitions,  together  with  an  account  of  receipts,  disburse- 
ments, and  transfers  to  miscellaneous  receipts,  under  this  authoriza- 
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Report  to  tion  shall  be  made  annually  in  connection  with  the  budget  estinu  I*    to 

Budget  Bureau         the  Director  of  the  Bureau  of  the  Budget  and  to  the  Congress,  and 
and  Congress.  the  inclusion  in  appropriation  acts  of  provisions  regulating  the  o] 

tion  of  the  automatic  data  processing  fund,  or  limiting  the  expendi- 
tures therefrom,  is  hereby  authorized. 
Appropriation.  "(d)   There  are  authorized  to  be  appropriated  to  said  fund  such 

sums  as  may  be  required  which,  together  with  the  value,  as  determined 
by  the  Administrator,  of  supplies  and  equipment  from  time  to  time 
transferred  to  the  Administrator;  shall  constitute  the  capital  of  the 
fund:  Provided^  That  said  fund  shall  bo  credited  with  (I)  i  Ivi 
and  reimbursements  from  available  appropriations  and  funds  of  any 
agency  (including  the  General  Services  Administration),  < 
tion,  or  contractor  utilizing  such  equipment  and  services  rendered 
them,  at  rates  determined  by  the  Administrator  to  approximate  the 
costs  thereof  met  by  the  fund  (including  depreciation  of  equipment, 
provision  for  acci  ued  have,  and  for  amortization  of  installation  c  ts, 
but  excluding,  in  the  determination  of  rates  prior  to  the  fiscal  year 
1 907,  such  direct  operating  expenses  as  may  be  directly  appropriated 
for,  which  expenses  may  be  charged  to  the  fund  and  covered  by 
advances  or  reimbursements  from  such  direct,  appropriations)  and 
(2)  refunds  or  recoveries  resulting  from  operations  of  the  fund, 
including  the  net  proceeds  of  disposal  of  excess  or  surplus  personal 
property  and  receipts  from  carriers  and  others  for  loss  of  o:  da) 
to  property  :  Provided  further,  That  following  the  close  of  each  ;  I 
year  any  net  income,  after  making  provisions  for  prior  year  losses, 
if  any,  shall  be  transferred  to  the  Treasury  of  the  United  States  as 
miscellaneous  receipts. 
"(e)  The  proviso  following  paragraph  (4)  in  section  201(a)  of  this 
63  stat.  384.  Act  and  the  provisions  of  section  602(d)  of  this  Act  shall  have  no  ap- 
40  use  481,  474.    plication  in  the  administration  of  this  section.    No  other  provi 

of  this  Act  or  any  other  Act  which  is  inconsistent  with  the  provisions 

of  this  section  shall  he  applicable  in  the  administration  of  this  section. 

Scientifio  and  "(f)   The  Secretary   of  Commerce    is  authorized    (J)    to   provide 

technological  agencies,  and  the  Administrator  of  General  Services  in  the  exen  ' 

advisory  serv-        0f  jjie  authority  delegated  in  this  section,  with  scientific  and  tech.no- 

oes  by  .eore-        logical  advisory  services  relating  to  automatic  data  processing  and 

tary   of  Coi.ir.erce.       7  ,     ,  ,         J         ,    ,n,     ,  f>  .    ,  '  ,    , . °        . 

related  systems,  and  (2)  to  make    appropriate  recommendations  to 

the  President  relating  to  the  establishment  of  uniform  Federal  auto- 
matic data  processing  standards.  The  Secretary  of  Commerce  is 
authorized  to  undertake  the  necessary  research  in  the  sciences  and 
technologies  of  automatic  data  processing  computer  and  related  sys- 
tems, as  may  be  required  under  provisions  of  this  subsection. 

"(g)  The.  authority  conferred  upon  the  Administrator  and  the 
Secretary  of  Commerce  by  this  section  shall  be  exercised  subject  to 
direction  by  the  President  and  to  fiscal  and  policy  control  exercised 
by  the  Bureau  of  the  Budget.  Authority  so  conferred  u]x>n  the 
Administrator  shall  not  be  so  construed  as  to  impair  or  interfere  with 
the  determination  by  agencies  of  their  individual  automatic  data  proc- 
essing equipment  requirements,  including  the  development  of  specifi- 
cations for  and  the  selection  of  the  types  and  configurations  of  equip- 
ment needed.  The  Administrator  shall  not  interfere  with,  or  attempt 
to  control  in  any  way,  the  use  made  of  automatic  data  processing 
equipment  or  components  thereof  by  any  agency.     The  Administra- 
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tor  shall  provide  adequate  notice  to  all  agencies  and  other  users  con-  Notice  "to 
ccrned  with  respect  to  each  proposed  determination  specifically  affect-  agenoies. 
ing  them  or  the  automatic  data  processing  equipment  or  components 
used  by  them.  In  the,  absence  of  mutual  agreement  between  the 
Administrator  and  the  agency  Ot-  user  concerned,  such  proposed  deter- 
minations shall  be  subject  to  review  and  decision  by  the  Bureau  of  the 
Budget,  unless  the  President  otherwise  directs." 

Approved  October  30,    1965. 
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Use  and  Management  of  Computer 
Technology 

The  President's  Memorandum  to  Heads  of  Departments 
and  Agencies.     June  28,  1906 

I  want  the  head  of  ever)'  Federal  agency  to  explore  and 
apply  all  possible  means  to 

-—use  the  electronic  computer  to  do  a  better  job 

—manage  computer  activity  at  the  lowest  possible  cost. 

I  want  my  administration  to  give  priority  emphasis  to 
both  of  these  objectives — nothing  less  will  suffice. 

The  electronic  computer  is  having  a  greater  impact  on 
what  the  Government  does  and  how  it  docs  it  than  any 
other  product  of  modern  technology. 

The  computer  is  making  it  possible  to 

— send  men  and  satellites  into  space 

—make  significant  strides  in  medical  research 

— add  several  billions  of  dollars  to  our  revenue  through 
improved  tax  administration 

- — administer  the  huge  and  complex  social  security  and 

'    medicare  programs 

—manage  a  multi-billion  dollar  defense  logistics  system 

- — speed  the  issuance  of  G.I.  insurance  dividends,  at 
much  less  cost 

- — save  lives  through  better  search  and  rescue  operations 

— harness  atomic  energy  for  peaceful  uses 

—design  better  but  less  costly  highways  and  structures. 

In  short,  computers  are  enabling  us  to  achieve  progress 
and  benefits  which  a  decade  ago  were  beyond  our  grasp. 

The  technology  is  available.  Its  potential  for  good 
has  been  amply  demonstrated,  but  it  remains  to  be  tapped 
in  fuller  measure. 

I  am  determined  that  we  take  advantage  of  this  tech- 
nology by  using  it  imaginatively  to  accomplish  worthwhile 
purposes. 

I  therefore  want  every  agency  head  to  give  thorough 
study  to  new  ways  in  which  the  electronic  computer 
might  be  used  to 

— provide  better  sei  ^  ice  to  the  public 

—  improve  agem  y  pcrformam  e 

—  reduce  ( osts. 

but,  as  we  use  computers  to  achieve  th  :sc  benefits,  1 
want  these  activities  managed  at  the  lowest  possible  cost. 
At  the  present  titne,  the  Federal  Government 
— uses  2,600  computers 
— employs  7  1,000  pi  i  ipl<  in  this  ai  lis  ity 
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— spends  over  $2  billion  annually  to  acquire  and  op- 
crate  this  equipment,  including  special  military  type 
computers. 
Clearly,  we  must  devote  our  best  efforts  to  managing 
this  large-  investment  wisely  and  with  the  least  cost. 

I  approved  a  blueprint  for  action  when  I  approved  the 
Bureau  of  the  Budget  "Report  on  Management  of  ADP 
in  the  Government." 

The  Congress  recognized  this  need  when  it  enacted 
Public  Law  89  306  (the  brooks  Bill)  last  October.  This 
legislation  provided  spe<  ifk  authorities  to 

— the  General  Services  Administration,  for  the  pro- 
curement, utilization  and  disposition  of  automatic 
data  processing  equipment 
— the  Department  of  Commerce,  for  the  development 
of  data  processing  standards  and  the  provision  of 
assistance  to  agencies  in  designing  computer-based 
systems 
— the  Bureau  of  the  Budget,  for  exercising  policy  and 
fiscal    control    over    the    implementation    of    these 
authorities. 
These  agencies  are  seeking  actively  to  put  into  effect 
ways  for  improving  and  reducing  the  cost  of  this  huge  and 
complex  operation. 

In  my  Budget  Message  for  1967  I  told  the  Congress 
of  my  intent  to  make  sine  that  this  huge  investment  is 
managed  efficiently. 

The  Federal  Government  must  give  priority  attention 
to 

— establishing  better  and  more  effective  procurement 

methods 
■ — making  fuller  use  of  existing  facilities  through,  shar- 
ing  and   joint-use   arrangements   before   acquiring 
additional  equipment 
— re-utilizing  excess  equipment  whenever  feasible 
— achieving,  with  industry  cooperation,  greater  com- 
patibility of  equipment. 
I  expect  all  agencies  to  cooperate  fully  with  the  Bureau 
of  the  Budget,  the  General  Services  Administration,  and 
the  Department  of  Commerce  hi  accomplishing  these 
objectives. 

I  want  the  Director  of  the  Bureau  of  the  Budget  to 
report  to  me  on  December  31,  1  966,  and  every  six  months 
thereafter,  on  the  progress  that  is  being  made  throughout 
the  Federal  Government  in  improving  the  management 
of  this  vei  y  impi  n  taut  tei  hnology. 

Lyndon  B.  Johnson 
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It  ..  my  cess rs  f.-.av  vne  Department  ov  Do-   ..-..  3  (De^j  rsc    only  concen- 
trate its  efforts  en  attainment  of  the  President's  objectives  but  provide 
z.r,  example  for  rhe  rest  ov  the  Uoverr.menr  v.   vol  lew.    Wn-le  .remer.oou< 
progress  ha    been  i    ::de  during  i'he  past  f:\     years,  !  believe  we  must 
increase  cur  .  f verts  to  achieve  more  effective  use  of  com;  ur:  ;s  in  all 

J?  •  r     -         -  ^  .|  .  ■  --i  r  •  j  •      «  i 

racers  ov  Lyeven v  _     lanagemenv  c  r.z.  c  !ercrions«     ine      ■esioer.v     j.iifs  oui 
incr  rners  are  2,ouu  cc;;^  urers  in  use  in  the  .  .:  zrc.i  (jcvemm        ana 
that  /I, 0^0  peopie  are  employed  vo  operate  v.-.::.:,     ..      ..  o.J  operates 
almost  2, LoJ  or  rnese  cempurers  ar.c;  employs  Dl,(  ^J  or  rne  rev   .  ;  ^::ir\- 
nel.    Wirn  tnss  large  snare  ov  rhe  [-ec  -.:.;  C^cvernmenv  mvesvm^  r:  v/e 
must  bear  a  large  snare  c;  rhe  respons:onr.y  .. 
direction  in  accordance  with  :!v3  ?ves:c'3:-.v:^  cbjecfives. 

I  have  reviewed  the  Fvc-!cv;.vv:c  objectives  end  curren;  DcD  yO.'.c'.z: 
which  eve  expressed  in  DoD  Directive  5100*40,  September  23,  1963, 
"F::ecr..,ioi  !ivie:  for  the  Administration  of  Automatic  Data  Processing 
Equipment  Program."    \  find  that  this  Directive  c!ree:;'y  ^vevie'e: 
sound  policies  within  which  the  fv^iiec.-..:s  cbjecfives  may  be  achieve' 

7 ever,  i      :  ess?"-. :c;i  vhav  we  increase  cur  £     :  ' . "   .  "  "ov 

use  of  computers .    ^e:e:.:e  mancc    _:  must  c:'"c:  '      ;■!  res      .  fne 

uses  and  c;     -bi      ies  of  [':.:  ccmpuier;     valuate  and  :'  !;■'  stc   e 

r: '/-  i  rem  :  " ;.  fos ;  ■  I  "    "    -mat"  Dn;  and  exe- 
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3.      We  must,  prior  to  computer  selection,  develop  :..,„  issue 
system  specifications  which  adequately  describe  the  systems  to  oe  per- 
formed  end  which  will  result  in  selection  of  computers  which  can 

i.    We  should  not  bo  forced 


....  compurers    ire- 


safisry  too  requirements  o.  that  specs vicatio 

to  acquire  additional  units  ct  later  dates  or 

maturely  because  ovsrlooticns  based  on  inadequate  system  specifications 

or  on  less  than  responsive  vendor  proposals.    Miiirc  ry  Departments  and 

DoD  agencies  should  centrally  prescribe  and  control  fhe  development  of 

sysrem  specifications  .or  co.v.puvor  acquisition  purposes. 


4.  We  most  make  competitive  computer  evaluation  and  selection 
a  professional  endeavor  and  organize  and  s::.!7:  yor  it  accordingly .  Staffs 
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Logistics)  (ASD  v----.  ''"°  begin  irnmedictely  c.  reviev/  and  evaluation 
or  the  overall  computer  management  program  in  the  DoD  and  to 
re co.-.: mend  po'lcy  changes  which  he  finds  are  necessary  :o  more  fully 
satisfy  the  ?;^z'.c.ar.'rs  objectives .    inso  doing  the  ASD  (.  J. L)  vvi ! ! : 

1  .  IrAer.y^y  hi:  conduct  of  selective  management  reviews  of 
automatic  data  ^:cc-:;!.:^  systems  and  installations  within  the  DcD. 
These  reviews  will  <"!ve  soee'e!  attention  to: 


a.      Refining  computer  sy: 


:  ns  opera-; 
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competitive  selection  of  conj:       ;rs  cr.c!  recommend  changes  that  may 
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anc  Acquisition  of  Automatic  Dcva  Processing  Equipment." 

5,      Evaluate  the  DoD  computer  mane;  : ment  information  system 
requirement  set  rorvn  in  UoU  Ins/ruction  7/w.o,  August  lU,    IVcO, 
"DoD  Automatic  Date    Processing  Equipment  Program  Reoorting  System," 
end  revise  It  as  necessary  to  insure  avci lability  o!  adequate  information 
ct  ecch  management  echelon  in  the  DcD, 

As  the  President  has  indicated ,  this  Department  wili,  of  course, 
cooperate  fully  v/itn  rne  .  jh  :  -  or  rhe  bucger,  rne  Department  ov 
Commerce  and  the  General  Service;  Administration  regarding  their 
programs  in  the  computer  area.    The  ASD  (i&L)  will  coordinate  these 
activities  ^cr  iae  DoD. 


The  President  has  provided  rhe  Department  cv  Defense  with  a  great 

III  r-  i 

full  cooperation 


challenge.    Ecch  of  ycu  should  give  this  matter  personc  !  attention  end 
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APPENDIX  E 


DIRECTIVES  GOVERNING  ACTIONS  OF  ADPESO 


Bureau  of  Budget  Circulars 

©  A- 27,     Policies  and  Responsibilities  of  the  Sharing  and  Electronic  Computer 

Time  and  Services  in  the  Executive  Branch 

o  A- 54,      Policies  on. Selection  and  Acquisition  of  Automatic  Data  Processing 

(ADP)  Equipment 

o  A- 61,      Guidelines  for  Appraising  Agency  Practices  in  the  Management  of 

Automatic  Data  Processing  (ADP)  Equipment  in  Federal  Agencies 

o  A- 94,      Discount  Rates  and  Procedures,  to  Be  Used  Evaluating  Deferred  Costs 

and  Benefits 


Department  of  Defense  Instructions 

©  4160. 19,      DOD  Automatic  Data  Processing  Equipment  Reutilization  Program, 

May8,   1969 

e  7041.  3,      Economic  Analysis  of  Proposed  Department  of  Defense  Investments, 

February  28,   1969 

o  5100.40,      Responsibilities  for  the  Administration  of  Automatic  Data  Processing 

Equipment  Program,   September  28,    1963 

©  4160.  1,      Nonexcess  Personal  Property  to  be  Sold  or  Exchanged  for  Replace- 

ment Purposes,  August  10,   1966 

©  4160.21,      Department  of  Defense  Personal  Property  Disposal  Program, 

April  28,   1967 
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(  4140.  34,      Department  of  Defense  Personal  Properly  Utilization  Program, 

September  5,   1968 


Department  of  the  Navy  Instructions 

g  Comptroller  of  the  Navy  Instruction  7000.28,  Economic  Analysis  of  Proposed 

Department  of  the  Navy  Investments,  April  19,   1967 

©  Secretary  of  the  Navy  Instruction  10462.  7B,  Department  of  the  Navy  Automatic 

Data  Processing  Program.  March  11,   1966 


APPENDIX  F 

AUTOMATIC  DATA  PROCESSING  SYSTEM  SPECIFICATIONS 

CONTENTS 

Part  I    -  Supporting  Information 

Section  I  -  Justification  for  Acquisition  of  ADP  System 

1-1    History 

1-2    System  Justification 

1-3    Budgeting  Estimates  and  Limitations 
Section  2  -  Workload  Description 

2-1    Representative  Programs 

2-2    Derivation  of  Representative  Programs 
Section  3  -  Explanation  of  Requirements 

3-1    Justification  of  Mandatory  Requirements 

3-2    Statement  of  Desirable  Features 

3-3    Statements  of  Value 

Part  II  -  ADP  System  Specifications 

Section  1  -  Format  and  Content  of  ADP  System  Specifications 

ATTACHMENT  I 

Chapter  1  -  Background  and  Operational  Plans 

1-1    Background 

1-2    Operational  Plans 
Chapter  2  -  Requirements 

2-1    System  Mandatory  Requirements 

2-2    Desirable  Features 
Chapter  3  -  Explanation  of  Terms  and  Abbreviations 

ATTACHMENT  II 

Chapter  1  -  System  Questionnaire 
Chapter  2  -  Timing  Information 

2-1    General  Information 

2-2    Program  Description 

2-3    Extension  Factors  and  Timing  Tables 

2-4    Benchmark  Demonstration  Instructions 
Section  2  -  Benchmark  Package 

SOURCE:    U.S.  ,  Department  of  the  Navy,  Specification,   Selection,  and 
Acquisition  of  Automatic  Data  Processing  Equipment,   Secretary  of  the  Navy. 
Instruction  10462.13,  Department  of  the  Navy,  Washington,  D.C.,  20390,  p.  B-2, 
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APPENDIX  G 


COMPUTER  SYSTEMS  IN  SCERT  LIBRARY 


EMR 
6000  Series 


Burroughs 
100  Series 
200  Series 
300  Scries 
2500 
3500 
5500 
6500 


Control  Data 

1 60A 

1604  A 

1700 

3100 

3200 

3300 

3400 

3500 

3600 

3800 

6400 

6500 

6600 


General  Electric 

115 

120 

205 

215 

225 

235 

415 

425 

435 

625 

635 


Honeywell 

110 

120 

125 

200 

400 

800 

1200 

1250 

1400 

1800 

2200 

3200 

4200 

8200 

IBM 

360/20 

360/25 

360/30 

360/40 

360/44 

360/50 

360/65 

360/67 

360/75 

360/85 

360/195 

1130 

1401 

1410 

1440 

1620 

7010 

7040  Series 

705 

7070  Series 

7080 

7090  Series 

NCR 

315 

3 1 5  RMC 

CENTURY-100 

CENTURY-200 

1CL 

1900  Series 
System  4  Series 


Digital  I  quipment 
PDP-  6 
PDP-  8 
PDP-  8i 
PDP- 10 


Phil  co 
2000  Series 

RCA 
301 

501 

601 

3301 

70/15 

70/25 

70/35 

70/45 

70/46 

70/55 

70/60 

UNI  VAC 

111 

418 

418-111 

490  Series 

1004  Series 

1005 

1050 

1  1 06 

1107 

1108 

9200 

9300 

9400 


SDS 

920 
925 
930 
9300 
Sigma  2 
Sigma  5 
Sigma  7 
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APPENDIX  H 


PRODUCTION  OF  OUTPUT  REPORTS 


SCERT  has  been  designed  to  produce  a  complete  series  of  "standard"  output 
reports  but  also  to  be  readily  adaptable  to  the  production  of  nonstandard  or  special 
purpose  reports.    The  reports  of  the  standard  set  are  all  programmed  as  optional, 
and  the  typical  use  of  the  program  calls  for  just  those  reports  needed  for  one's 
particular  analysis.    A  brief  description  of  the  standard  reports: 

Computer  Complement  Report  -  Portrays  the  exact  configuration  that  was 
simulated  and  provides  certain  basic  cost  data  about  that  configuration. 

Central  Processor  Utilization  -  Summarizes  for  each  scheduled  run  the 
projected  running  time,   set-up  time,  and  horizontal  memory  utilization. 

Programming  Requirements  -  Projects  for  each  run  an  implementation- oriented 
by-product  of  the  simulation  by  summarizing  the  number  of  steps  to  be 
programmed  and  an  estimate  of  the  number  of  programmer  man  months 
required. 

Application  Summary  -  If  the  processing  requirements  simulated  can  be 
broken  into  discreet  application  areas,  this  report  summarizes  running- 
time  and  programming  effort  for  each  of  these  areas. 

Computer  Capabilities  -  Tabulates  for  each  scheduled  run  simulated  break- 
out of  thru- put  requirements  and  then  a  total  for  the  entire  simulation.     It  is 
used  primarily  to  pinpoint  critical  hardware  areas. 

Cost  Summary  -  Primary  cost  output  which  relates  the  projected  utilization  of 
the  compute]"  to  lease,  purchase,  and  maintenance  cost  considerations. 
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Real-time  Analysis  -  If  random  real-time  processing  was  involved,  a  series  of 
reports  are  produced  to  portray  the  impact  of  such  processing  on  the  computer 
and  the  response  expected  of  the  computer  for  such  processing.    The  four 
standard  reports  are: 

Event  Processing  Analysis  -  Analyzes  each  random  event  in  terms  of 
its  unique  thru- put. 

Hardware  Utilization  -  Analyzes  every  potential  queue  present  in  the 
hardware  complex. 

Systems  Response  -  Reflects  the  expected  95th  and  99th  percentile  of 
computer  and  communications  network  response. 

Memory  Requirements  -  Outlines  the  unique  memory  required  for  each 
random  event  and  the  expected  and  worst  case  background  requirements. 

Multiprogram  Run  Schedule  -  Produced  whenever  multiprogramming  has  been 
simulated  and  reflects  those  runs  which  have  been  scheduled  by  the  simulator  to 
operate  concurrently. 

Detail  Analysis  -  Produced  for  each  scheduled  run  or  real-time  event  simulated 
and  serves  as  backup  for  all  other  reports.     It  precisely  portrays  the  utilization, 
timing,  and  memory  requirements  derived  by  the  simulation  and  presimulation 
algorithms  for  all  components  making  up  the  run. 

Summary  Analysis  -  Summarizes,  for  each  configuration  simulated,  the 
performance  and  cost/performance  derived. 

System  Documentation  -  A  by-product  report  which  produces  a  standard  form  of 
documentation  of  all  processing  requirements.     It  is  especially  useful  in  securing 
responsive,   realistic  hardware  bids  from  computer  manufacturers. 


APPENDIX  J 
SAMPLE  FORMAT  FOR  DISCOUNTING  DEFERRED  COSTS  AND  BENEFITS 


Assume  a  ten-year  program  which  will  commit  the  Government  to 
the  stream  of  expenditures  appearing  in  column  (2)  of  the 
table  below  and  v/hich  will  result  in  a  series  of  benefits 
appearing  in  column  (3)  .   The  discount  factor  for  a  10  percent- 
discount  rate  is  presented  in  column  (4).   Present  value  cost 
for  each  of  the  ten  years  is  calculated  by  multiplying 
column  (2)  by  column  (4);  present  value  benefit  for  each  of 
the  ten  years  is  calculated  by  multiplying  column  (3)  by 
column  (4).   Present  value  costs  and  benefits  are  presented 
in  columns  (5)  and  (6),  respectively. 

Present  Present 
Value       Value 

Expected  Expected   Discount     Cost  Benefit 

Year  of    Yearly    Yearly   Factor  for  [Col.(2)x  [Col.(3)x 

Operation    Cost    Benefit   10  Percent   Col. (4) ]  Col . (4) ] 

(1)       "  (2)       (3)         (4)  (5)  (6) 

1  $10      $  0       0  .909       $9.1       $  0.0 

2  20  0  0-826  16.5  0.0 

3  30  5  0-751  22.5  3.8 

4  30  10  0-683  20.5  6.8 

5  20  30  0-621  12.4  18.6 

6  10  40  0-564  5.6  22.6 

7  5  4  0  0.513  2.6  20.5 

8  5  40  0.467  2.3  18.7 

9  5  4  0.  0.4  24  2.1  17.0 
10  5  25  0.386  1.9  9.7 


$95.5      $117.7 

The  sum  of  column  (5)  is  present  value  cost:   $95.5. 

The  sum  of  column  (6)  is  present  value  benefit:   $117.7. 

Present  value  net  benefit  is  the  difference  between  present 
value  total  benefit  and  present  value  total  cost: 
$117.7  -  $95.5  -  $22.2. 

The  benefit-cost  ratio  is  117.7/9  5.5  -  1.23. 

NOTE:   For  more  difficult  discounting  problems,  a  recom- 
mended reference  is  Principles  o.f .  .Engineering 
Economy,  by  Eugene  L.  Grant  a:\d   W.  G.  Ireson, 
Ronald  Press  Company,  1960. 
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